You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "Jackie-Jiang (via GitHub)" <gi...@apache.org> on 2024/04/07 23:36:06 UTC
[PR] Enhance Kinesis consumer [pinot]
Jackie-Jiang opened a new pull request, #12806:
URL: https://github.com/apache/pinot/pull/12806
- Do not use a separate thread to fetch Kinesis records (this can fix the potential race condition)
- Follow the same convention with Kafka:
- Return the non-empty message batch immediately
- Wait up to timeout if there is no record fetched, then return an empty message batch
- Change back the default timeout to 5s
- Change the default max records per fetch to 10,000 (Kinesis default)
- Remove some unused dependencies
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Enhance Kinesis consumer [pinot]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #12806:
URL: https://github.com/apache/pinot/pull/12806#issuecomment-2041658423
## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/12806?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
Attention: Patch coverage is `0%` with `41 lines` in your changes are missing coverage. Please review.
> Project coverage is 0.00%. Comparing base [(`59551e4`)](https://app.codecov.io/gh/apache/pinot/commit/59551e45224f1535c4863fd577622b37366ccc97?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to head [(`8d1a28f`)](https://app.codecov.io/gh/apache/pinot/pull/12806?dropdown=coverage&src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
> Report is 219 commits behind head on master.
| [Files](https://app.codecov.io/gh/apache/pinot/pull/12806?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...e/pinot/plugin/stream/kinesis/KinesisConsumer.java](https://app.codecov.io/gh/apache/pinot/pull/12806?src=pr&el=tree&filepath=pinot-plugins%2Fpinot-stream-ingestion%2Fpinot-kinesis%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fplugin%2Fstream%2Fkinesis%2FKinesisConsumer.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtcGx1Z2lucy9waW5vdC1zdHJlYW0taW5nZXN0aW9uL3Bpbm90LWtpbmVzaXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3BsdWdpbi9zdHJlYW0va2luZXNpcy9LaW5lc2lzQ29uc3VtZXIuamF2YQ==) | 0.00% | [39 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/12806?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...java/org/apache/pinot/spi/stream/StreamConfig.java](https://app.codecov.io/gh/apache/pinot/pull/12806?src=pr&el=tree&filepath=pinot-spi%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fspi%2Fstream%2FStreamConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3Qtc3BpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcGkvc3RyZWFtL1N0cmVhbUNvbmZpZy5qYXZh) | 0.00% | [2 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/12806?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #12806 +/- ##
=============================================
- Coverage 61.75% 0.00% -61.76%
=============================================
Files 2436 2387 -49
Lines 133233 131080 -2153
Branches 20636 20269 -367
=============================================
- Hits 82274 0 -82274
- Misses 44911 131080 +86169
+ Partials 6048 0 -6048
```
| [Flag](https://app.codecov.io/gh/apache/pinot/pull/12806/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [custom-integration1](https://app.codecov.io/gh/apache/pinot/pull/12806/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration](https://app.codecov.io/gh/apache/pinot/pull/12806/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <0.00%> (-0.01%)` | :arrow_down: |
| [integration1](https://app.codecov.io/gh/apache/pinot/pull/12806/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration2](https://app.codecov.io/gh/apache/pinot/pull/12806/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <0.00%> (ø)` | |
| [java-11](https://app.codecov.io/gh/apache/pinot/pull/12806/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [java-21](https://app.codecov.io/gh/apache/pinot/pull/12806/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <0.00%> (-61.63%)` | :arrow_down: |
| [skip-bytebuffers-false](https://app.codecov.io/gh/apache/pinot/pull/12806/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [skip-bytebuffers-true](https://app.codecov.io/gh/apache/pinot/pull/12806/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <0.00%> (-27.73%)` | :arrow_down: |
| [temurin](https://app.codecov.io/gh/apache/pinot/pull/12806/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <0.00%> (-61.76%)` | :arrow_down: |
| [unittests](https://app.codecov.io/gh/apache/pinot/pull/12806/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [unittests1](https://app.codecov.io/gh/apache/pinot/pull/12806/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [unittests2](https://app.codecov.io/gh/apache/pinot/pull/12806/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/pinot/pull/12806?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Enhance Kinesis consumer [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on code in PR #12806:
URL: https://github.com/apache/pinot/pull/12806#discussion_r1556717366
##########
pinot-plugins/pinot-stream-ingestion/pinot-kinesis/src/main/java/org/apache/pinot/plugin/stream/kinesis/KinesisConsumer.java:
##########
@@ -69,134 +61,77 @@ public KinesisConsumer(KinesisConfig config, KinesisClient kinesisClient) {
super(config, kinesisClient);
}
- /**
- * Fetch records from the Kinesis stream between the start and end KinesisCheckpoint
- */
@Override
- public KinesisMessageBatch fetchMessages(StreamPartitionMsgOffset startMsgOffset, int timeoutMs) {
+ public synchronized KinesisMessageBatch fetchMessages(StreamPartitionMsgOffset startMsgOffset, int timeoutMs) {
KinesisPartitionGroupOffset startOffset = (KinesisPartitionGroupOffset) startMsgOffset;
- List<BytesStreamMessage> messages = new ArrayList<>();
- Future<KinesisMessageBatch> kinesisFetchResultFuture =
- _executorService.submit(() -> getResult(startOffset, messages));
- try {
- return kinesisFetchResultFuture.get(timeoutMs, TimeUnit.MILLISECONDS);
- } catch (TimeoutException e) {
- kinesisFetchResultFuture.cancel(true);
- } catch (Exception e) {
- // Ignored
- }
- return buildKinesisMessageBatch(startOffset, messages, false);
- }
-
- private KinesisMessageBatch getResult(KinesisPartitionGroupOffset startOffset, List<BytesStreamMessage> messages) {
- try {
- String shardId = startOffset.getShardId();
- String shardIterator = getShardIterator(shardId, startOffset.getSequenceNumber());
- boolean endOfShard = false;
- long currentWindow = System.currentTimeMillis() / SLEEP_TIME_BETWEEN_REQUESTS;
- int currentWindowRequests = 0;
- while (shardIterator != null) {
- GetRecordsRequest getRecordsRequest = GetRecordsRequest.builder().shardIterator(shardIterator).build();
- long requestSentTime = System.currentTimeMillis() / 1000;
- GetRecordsResponse getRecordsResponse = _kinesisClient.getRecords(getRecordsRequest);
- List<Record> records = getRecordsResponse.records();
- if (!records.isEmpty()) {
- for (Record record : records) {
- messages.add(extractStreamMessage(record, shardId));
- }
- if (messages.size() >= _config.getNumMaxRecordsToFetch()) {
- break;
- }
- }
-
- if (getRecordsResponse.hasChildShards() && !getRecordsResponse.childShards().isEmpty()) {
- //This statement returns true only when end of current shard has reached.
- // hasChildShards only checks if the childShard is null and is a valid instance.
- endOfShard = true;
- break;
- }
-
- shardIterator = getRecordsResponse.nextShardIterator();
-
- if (Thread.interrupted()) {
- break;
- }
-
- // Kinesis enforces a limit of 5 .getRecords request per second on each shard from AWS end
- // Beyond this limit we start getting ProvisionedThroughputExceededException which affect the ingestion
- if (requestSentTime == currentWindow) {
- currentWindowRequests++;
- } else if (requestSentTime > currentWindow) {
- currentWindow = requestSentTime;
- currentWindowRequests = 0;
- }
-
- if (currentWindowRequests >= _config.getNumMaxRecordsToFetch()) {
- try {
- Thread.sleep(SLEEP_TIME_BETWEEN_REQUESTS);
- } catch (InterruptedException e) {
- LOGGER.debug("Sleep interrupted while rate limiting Kinesis requests", e);
- break;
- }
- }
- }
-
- return buildKinesisMessageBatch(startOffset, messages, endOfShard);
- } catch (IllegalStateException e) {
- debugOrLogWarning("Illegal state exception, connection is broken", e);
- } catch (ProvisionedThroughputExceededException e) {
- debugOrLogWarning("The request rate for the stream is too high", e);
- } catch (ExpiredIteratorException e) {
- debugOrLogWarning("ShardIterator expired while trying to fetch records", e);
- } catch (ResourceNotFoundException | InvalidArgumentException e) {
- // aws errors
- LOGGER.error("Encountered AWS error while attempting to fetch records", e);
- } catch (KinesisException e) {
- debugOrLogWarning("Encountered unknown unrecoverable AWS exception", e);
- throw new RuntimeException(e);
- } catch (AbortedException e) {
- if (!(e.getCause() instanceof InterruptedException)) {
- debugOrLogWarning("Task aborted due to exception", e);
- }
- } catch (Throwable e) {
- // non transient errors
- LOGGER.error("Unknown fetchRecords exception", e);
- throw new RuntimeException(e);
- }
- return buildKinesisMessageBatch(startOffset, messages, false);
- }
+ String shardId = startOffset.getShardId();
+ String startSequenceNumber = startOffset.getSequenceNumber();
- private void debugOrLogWarning(String message, Throwable throwable) {
- if (LOGGER.isDebugEnabled()) {
- LOGGER.debug(message, throwable);
+ // Get the shard iterator
+ String shardIterator;
+ if (startSequenceNumber.equals(_nextStartSequenceNumber)) {
+ shardIterator = _nextShardIterator;
} else {
- LOGGER.warn(message + ": " + throwable.getMessage());
+ // TODO: Revisit the offset handling logic. Reading after the start sequence number can lose the first message
+ // when consuming from a new partition because the initial start sequence number is inclusive.
+ GetShardIteratorRequest getShardIteratorRequest =
+ GetShardIteratorRequest.builder().streamName(_config.getStreamTopicName()).shardId(shardId)
+ .startingSequenceNumber(startSequenceNumber).shardIteratorType(ShardIteratorType.AFTER_SEQUENCE_NUMBER)
+ .build();
+ shardIterator = _kinesisClient.getShardIterator(getShardIteratorRequest).shardIterator();
+ }
+ if (shardIterator == null) {
+ return new KinesisMessageBatch(List.of(), startOffset, true);
}
- }
- private KinesisMessageBatch buildKinesisMessageBatch(KinesisPartitionGroupOffset startOffset,
- List<BytesStreamMessage> messages, boolean endOfShard) {
+ // Read records
+ rateLimitRequests();
+ GetRecordsRequest getRecordRequest =
+ GetRecordsRequest.builder().shardIterator(shardIterator).limit(_config.getNumMaxRecordsToFetch()).build();
+ GetRecordsResponse getRecordsResponse = _kinesisClient.getRecords(getRecordRequest);
+ List<Record> records = getRecordsResponse.records();
+ List<BytesStreamMessage> messages;
KinesisPartitionGroupOffset offsetOfNextBatch;
- if (messages.isEmpty()) {
- offsetOfNextBatch = startOffset;
- } else {
+ if (!records.isEmpty()) {
+ messages = records.stream().map(record -> extractStreamMessage(record, shardId)).collect(Collectors.toList());
StreamMessageMetadata lastMessageMetadata = messages.get(messages.size() - 1).getMetadata();
assert lastMessageMetadata != null;
offsetOfNextBatch = (KinesisPartitionGroupOffset) lastMessageMetadata.getNextOffset();
+ } else {
+ // TODO: Revisit whether Kinesis can return empty batch when there are available records. The consumer cna handle
+ // empty message batch, but it will treat it as fully caught up.
+ messages = List.of();
+ offsetOfNextBatch = startOffset;
}
- return new KinesisMessageBatch(messages, offsetOfNextBatch, endOfShard);
+ assert offsetOfNextBatch != null;
+ _nextStartSequenceNumber = offsetOfNextBatch.getSequenceNumber();
+ _nextShardIterator = getRecordsResponse.nextShardIterator();
+ return new KinesisMessageBatch(messages, offsetOfNextBatch, _nextShardIterator == null);
}
- private String getShardIterator(String shardId, String sequenceNumber) {
- GetShardIteratorRequest.Builder requestBuilder =
- GetShardIteratorRequest.builder().streamName(_config.getStreamTopicName()).shardId(shardId);
- if (sequenceNumber != null) {
- requestBuilder = requestBuilder.startingSequenceNumber(sequenceNumber)
- .shardIteratorType(ShardIteratorType.AFTER_SEQUENCE_NUMBER);
+ /**
+ * Kinesis enforces a limit of 5 getRecords request per second on each shard from AWS end, beyond which we start
+ * getting {@link ProvisionedThroughputExceededException}. Rate limit the requests to avoid this.
+ */
+ private void rateLimitRequests() {
Review Comment:
Kinesis [limit](https://docs.aws.amazon.com/streams/latest/dev/service-sizes-and-limits.html) is not very straight forward, so I guess we need to iterate on this to get the best settings.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Enhance Kinesis consumer [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on code in PR #12806:
URL: https://github.com/apache/pinot/pull/12806#discussion_r1556482317
##########
pinot-plugins/pinot-stream-ingestion/pinot-kinesis/src/main/java/org/apache/pinot/plugin/stream/kinesis/KinesisConsumer.java:
##########
@@ -69,113 +60,56 @@ public KinesisConsumer(KinesisConfig config, KinesisClient kinesisClient) {
super(config, kinesisClient);
}
- /**
- * Fetch records from the Kinesis stream between the start and end KinesisCheckpoint
- */
@Override
- public KinesisMessageBatch fetchMessages(StreamPartitionMsgOffset startMsgOffset, int timeoutMs) {
+ public synchronized KinesisMessageBatch fetchMessages(StreamPartitionMsgOffset startMsgOffset, int timeoutMs) {
KinesisPartitionGroupOffset startOffset = (KinesisPartitionGroupOffset) startMsgOffset;
- List<BytesStreamMessage> messages = new ArrayList<>();
- Future<KinesisMessageBatch> kinesisFetchResultFuture =
- _executorService.submit(() -> getResult(startOffset, messages));
- try {
- return kinesisFetchResultFuture.get(timeoutMs, TimeUnit.MILLISECONDS);
- } catch (TimeoutException e) {
- kinesisFetchResultFuture.cancel(true);
- } catch (Exception e) {
- // Ignored
- }
- return buildKinesisMessageBatch(startOffset, messages, false);
- }
-
- private KinesisMessageBatch getResult(KinesisPartitionGroupOffset startOffset, List<BytesStreamMessage> messages) {
- try {
- String shardId = startOffset.getShardId();
- String shardIterator = getShardIterator(shardId, startOffset.getSequenceNumber());
- boolean endOfShard = false;
- long currentWindow = System.currentTimeMillis() / SLEEP_TIME_BETWEEN_REQUESTS;
- int currentWindowRequests = 0;
- while (shardIterator != null) {
- GetRecordsRequest getRecordsRequest = GetRecordsRequest.builder().shardIterator(shardIterator).build();
- long requestSentTime = System.currentTimeMillis() / 1000;
- GetRecordsResponse getRecordsResponse = _kinesisClient.getRecords(getRecordsRequest);
- List<Record> records = getRecordsResponse.records();
- if (!records.isEmpty()) {
- for (Record record : records) {
- messages.add(extractStreamMessage(record, shardId));
- }
- if (messages.size() >= _config.getNumMaxRecordsToFetch()) {
- break;
- }
+ String shardId = startOffset.getShardId();
+ String startSequenceNumber = startOffset.getSequenceNumber();
+
+ // NOTE: Kinesis enforces a limit of 5 getRecords request per second on each shard from AWS end, beyond which we
+ // start getting ProvisionedThroughputExceededException. Rate limit the requests to avoid this.
+ long currentTimeMs = System.currentTimeMillis();
+ int currentTimeSeconds = (int) TimeUnit.MILLISECONDS.toSeconds(currentTimeMs);
+ if (currentTimeSeconds == _currentSecond) {
+ if (_numRequestsInCurrentSecond == _config.getRpsLimit()) {
+ try {
+ Thread.sleep(1000 - (currentTimeMs % 1000));
+ } catch (InterruptedException e) {
+ throw new RuntimeException(e);
}
-
- if (getRecordsResponse.hasChildShards() && !getRecordsResponse.childShards().isEmpty()) {
- //This statement returns true only when end of current shard has reached.
- // hasChildShards only checks if the childShard is null and is a valid instance.
- endOfShard = true;
- break;
- }
-
- shardIterator = getRecordsResponse.nextShardIterator();
-
- if (Thread.interrupted()) {
- break;
- }
-
- // Kinesis enforces a limit of 5 .getRecords request per second on each shard from AWS end
- // Beyond this limit we start getting ProvisionedThroughputExceededException which affect the ingestion
- if (requestSentTime == currentWindow) {
- currentWindowRequests++;
- } else if (requestSentTime > currentWindow) {
- currentWindow = requestSentTime;
- currentWindowRequests = 0;
- }
-
- if (currentWindowRequests >= _config.getNumMaxRecordsToFetch()) {
- try {
- Thread.sleep(SLEEP_TIME_BETWEEN_REQUESTS);
- } catch (InterruptedException e) {
- LOGGER.debug("Sleep interrupted while rate limiting Kinesis requests", e);
- break;
- }
- }
- }
-
- return buildKinesisMessageBatch(startOffset, messages, endOfShard);
- } catch (IllegalStateException e) {
- debugOrLogWarning("Illegal state exception, connection is broken", e);
- } catch (ProvisionedThroughputExceededException e) {
- debugOrLogWarning("The request rate for the stream is too high", e);
- } catch (ExpiredIteratorException e) {
- debugOrLogWarning("ShardIterator expired while trying to fetch records", e);
- } catch (ResourceNotFoundException | InvalidArgumentException e) {
- // aws errors
- LOGGER.error("Encountered AWS error while attempting to fetch records", e);
- } catch (KinesisException e) {
- debugOrLogWarning("Encountered unknown unrecoverable AWS exception", e);
- throw new RuntimeException(e);
- } catch (AbortedException e) {
- if (!(e.getCause() instanceof InterruptedException)) {
- debugOrLogWarning("Task aborted due to exception", e);
+ _currentSecond++;
+ _numRequestsInCurrentSecond = 1;
+ } else {
+ _numRequestsInCurrentSecond++;
}
- } catch (Throwable e) {
- // non transient errors
- LOGGER.error("Unknown fetchRecords exception", e);
- throw new RuntimeException(e);
+ } else {
+ _currentSecond = currentTimeSeconds;
+ _numRequestsInCurrentSecond = 1;
}
- return buildKinesisMessageBatch(startOffset, messages, false);
- }
- private void debugOrLogWarning(String message, Throwable throwable) {
- if (LOGGER.isDebugEnabled()) {
- LOGGER.debug(message, throwable);
+ // Get the shard iterator
+ String shardIterator;
+ if (startSequenceNumber.equals(_nextStartSequenceNumber)) {
+ shardIterator = _nextShardIterator;
} else {
- LOGGER.warn(message + ": " + throwable.getMessage());
+ // TODO: Revisit this logic to see if we always miss the first message when consuming from a new shard
+ GetShardIteratorRequest getShardIteratorRequest =
+ GetShardIteratorRequest.builder().streamName(_config.getStreamTopicName()).shardId(shardId)
+ .startingSequenceNumber(startSequenceNumber).shardIteratorType(ShardIteratorType.AFTER_SEQUENCE_NUMBER)
+ .build();
+ shardIterator = _kinesisClient.getShardIterator(getShardIteratorRequest).shardIterator();
+ }
+ if (shardIterator == null) {
+ return new KinesisMessageBatch(List.of(), startOffset, true);
}
- }
- private KinesisMessageBatch buildKinesisMessageBatch(KinesisPartitionGroupOffset startOffset,
- List<BytesStreamMessage> messages, boolean endOfShard) {
+ // Read records
+ GetRecordsRequest getRecordRequest =
+ GetRecordsRequest.builder().shardIterator(shardIterator).limit(_config.getNumMaxRecordsToFetch()).build();
+ GetRecordsResponse getRecordsResponse = _kinesisClient.getRecords(getRecordRequest);
Review Comment:
We need some test to verify the behavior. The consumer can handle empty message batch, but the consumption lag might be set to 0 because it thought there is no more message. Added a TODO to revisit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Enhance Kinesis consumer [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on code in PR #12806:
URL: https://github.com/apache/pinot/pull/12806#discussion_r1556472410
##########
pinot-plugins/pinot-stream-ingestion/pinot-kinesis/src/main/java/org/apache/pinot/plugin/stream/kinesis/KinesisConsumer.java:
##########
@@ -69,113 +60,56 @@ public KinesisConsumer(KinesisConfig config, KinesisClient kinesisClient) {
super(config, kinesisClient);
}
- /**
- * Fetch records from the Kinesis stream between the start and end KinesisCheckpoint
- */
@Override
- public KinesisMessageBatch fetchMessages(StreamPartitionMsgOffset startMsgOffset, int timeoutMs) {
+ public synchronized KinesisMessageBatch fetchMessages(StreamPartitionMsgOffset startMsgOffset, int timeoutMs) {
KinesisPartitionGroupOffset startOffset = (KinesisPartitionGroupOffset) startMsgOffset;
- List<BytesStreamMessage> messages = new ArrayList<>();
- Future<KinesisMessageBatch> kinesisFetchResultFuture =
- _executorService.submit(() -> getResult(startOffset, messages));
- try {
- return kinesisFetchResultFuture.get(timeoutMs, TimeUnit.MILLISECONDS);
- } catch (TimeoutException e) {
- kinesisFetchResultFuture.cancel(true);
- } catch (Exception e) {
- // Ignored
- }
- return buildKinesisMessageBatch(startOffset, messages, false);
- }
-
- private KinesisMessageBatch getResult(KinesisPartitionGroupOffset startOffset, List<BytesStreamMessage> messages) {
- try {
- String shardId = startOffset.getShardId();
- String shardIterator = getShardIterator(shardId, startOffset.getSequenceNumber());
- boolean endOfShard = false;
- long currentWindow = System.currentTimeMillis() / SLEEP_TIME_BETWEEN_REQUESTS;
- int currentWindowRequests = 0;
- while (shardIterator != null) {
- GetRecordsRequest getRecordsRequest = GetRecordsRequest.builder().shardIterator(shardIterator).build();
- long requestSentTime = System.currentTimeMillis() / 1000;
- GetRecordsResponse getRecordsResponse = _kinesisClient.getRecords(getRecordsRequest);
- List<Record> records = getRecordsResponse.records();
- if (!records.isEmpty()) {
- for (Record record : records) {
- messages.add(extractStreamMessage(record, shardId));
- }
- if (messages.size() >= _config.getNumMaxRecordsToFetch()) {
- break;
- }
+ String shardId = startOffset.getShardId();
+ String startSequenceNumber = startOffset.getSequenceNumber();
+
+ // NOTE: Kinesis enforces a limit of 5 getRecords request per second on each shard from AWS end, beyond which we
+ // start getting ProvisionedThroughputExceededException. Rate limit the requests to avoid this.
+ long currentTimeMs = System.currentTimeMillis();
Review Comment:
I didn't find one from Kinesis client. Seems it will just throw `LimitExceededException`.
The `rps` is currently configured on Pinot side though, so I guess it makes sense to rate limit on the Pinot side.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Enhance Kinesis consumer [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on code in PR #12806:
URL: https://github.com/apache/pinot/pull/12806#discussion_r1556480965
##########
pinot-plugins/pinot-stream-ingestion/pinot-kinesis/src/main/java/org/apache/pinot/plugin/stream/kinesis/KinesisConsumer.java:
##########
@@ -69,113 +60,56 @@ public KinesisConsumer(KinesisConfig config, KinesisClient kinesisClient) {
super(config, kinesisClient);
}
- /**
- * Fetch records from the Kinesis stream between the start and end KinesisCheckpoint
- */
@Override
- public KinesisMessageBatch fetchMessages(StreamPartitionMsgOffset startMsgOffset, int timeoutMs) {
+ public synchronized KinesisMessageBatch fetchMessages(StreamPartitionMsgOffset startMsgOffset, int timeoutMs) {
KinesisPartitionGroupOffset startOffset = (KinesisPartitionGroupOffset) startMsgOffset;
- List<BytesStreamMessage> messages = new ArrayList<>();
- Future<KinesisMessageBatch> kinesisFetchResultFuture =
- _executorService.submit(() -> getResult(startOffset, messages));
- try {
- return kinesisFetchResultFuture.get(timeoutMs, TimeUnit.MILLISECONDS);
- } catch (TimeoutException e) {
- kinesisFetchResultFuture.cancel(true);
- } catch (Exception e) {
- // Ignored
- }
- return buildKinesisMessageBatch(startOffset, messages, false);
- }
-
- private KinesisMessageBatch getResult(KinesisPartitionGroupOffset startOffset, List<BytesStreamMessage> messages) {
- try {
- String shardId = startOffset.getShardId();
- String shardIterator = getShardIterator(shardId, startOffset.getSequenceNumber());
- boolean endOfShard = false;
- long currentWindow = System.currentTimeMillis() / SLEEP_TIME_BETWEEN_REQUESTS;
- int currentWindowRequests = 0;
- while (shardIterator != null) {
- GetRecordsRequest getRecordsRequest = GetRecordsRequest.builder().shardIterator(shardIterator).build();
- long requestSentTime = System.currentTimeMillis() / 1000;
- GetRecordsResponse getRecordsResponse = _kinesisClient.getRecords(getRecordsRequest);
- List<Record> records = getRecordsResponse.records();
- if (!records.isEmpty()) {
- for (Record record : records) {
- messages.add(extractStreamMessage(record, shardId));
- }
- if (messages.size() >= _config.getNumMaxRecordsToFetch()) {
- break;
- }
+ String shardId = startOffset.getShardId();
+ String startSequenceNumber = startOffset.getSequenceNumber();
+
+ // NOTE: Kinesis enforces a limit of 5 getRecords request per second on each shard from AWS end, beyond which we
+ // start getting ProvisionedThroughputExceededException. Rate limit the requests to avoid this.
+ long currentTimeMs = System.currentTimeMillis();
+ int currentTimeSeconds = (int) TimeUnit.MILLISECONDS.toSeconds(currentTimeMs);
+ if (currentTimeSeconds == _currentSecond) {
+ if (_numRequestsInCurrentSecond == _config.getRpsLimit()) {
+ try {
+ Thread.sleep(1000 - (currentTimeMs % 1000));
+ } catch (InterruptedException e) {
+ throw new RuntimeException(e);
}
-
- if (getRecordsResponse.hasChildShards() && !getRecordsResponse.childShards().isEmpty()) {
- //This statement returns true only when end of current shard has reached.
- // hasChildShards only checks if the childShard is null and is a valid instance.
- endOfShard = true;
- break;
- }
-
- shardIterator = getRecordsResponse.nextShardIterator();
-
- if (Thread.interrupted()) {
- break;
- }
-
- // Kinesis enforces a limit of 5 .getRecords request per second on each shard from AWS end
- // Beyond this limit we start getting ProvisionedThroughputExceededException which affect the ingestion
- if (requestSentTime == currentWindow) {
- currentWindowRequests++;
- } else if (requestSentTime > currentWindow) {
- currentWindow = requestSentTime;
- currentWindowRequests = 0;
- }
-
- if (currentWindowRequests >= _config.getNumMaxRecordsToFetch()) {
- try {
- Thread.sleep(SLEEP_TIME_BETWEEN_REQUESTS);
- } catch (InterruptedException e) {
- LOGGER.debug("Sleep interrupted while rate limiting Kinesis requests", e);
- break;
- }
- }
- }
-
- return buildKinesisMessageBatch(startOffset, messages, endOfShard);
- } catch (IllegalStateException e) {
- debugOrLogWarning("Illegal state exception, connection is broken", e);
- } catch (ProvisionedThroughputExceededException e) {
- debugOrLogWarning("The request rate for the stream is too high", e);
- } catch (ExpiredIteratorException e) {
- debugOrLogWarning("ShardIterator expired while trying to fetch records", e);
- } catch (ResourceNotFoundException | InvalidArgumentException e) {
- // aws errors
- LOGGER.error("Encountered AWS error while attempting to fetch records", e);
- } catch (KinesisException e) {
- debugOrLogWarning("Encountered unknown unrecoverable AWS exception", e);
- throw new RuntimeException(e);
- } catch (AbortedException e) {
- if (!(e.getCause() instanceof InterruptedException)) {
- debugOrLogWarning("Task aborted due to exception", e);
+ _currentSecond++;
+ _numRequestsInCurrentSecond = 1;
+ } else {
+ _numRequestsInCurrentSecond++;
}
- } catch (Throwable e) {
- // non transient errors
- LOGGER.error("Unknown fetchRecords exception", e);
- throw new RuntimeException(e);
+ } else {
+ _currentSecond = currentTimeSeconds;
+ _numRequestsInCurrentSecond = 1;
}
- return buildKinesisMessageBatch(startOffset, messages, false);
- }
- private void debugOrLogWarning(String message, Throwable throwable) {
- if (LOGGER.isDebugEnabled()) {
- LOGGER.debug(message, throwable);
+ // Get the shard iterator
+ String shardIterator;
+ if (startSequenceNumber.equals(_nextStartSequenceNumber)) {
+ shardIterator = _nextShardIterator;
} else {
- LOGGER.warn(message + ": " + throwable.getMessage());
+ // TODO: Revisit this logic to see if we always miss the first message when consuming from a new shard
Review Comment:
Ack
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Enhance Kinesis consumer [pinot]
Posted by "KKcorps (via GitHub)" <gi...@apache.org>.
KKcorps merged PR #12806:
URL: https://github.com/apache/pinot/pull/12806
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Enhance Kinesis consumer [pinot]
Posted by "swaminathanmanish (via GitHub)" <gi...@apache.org>.
swaminathanmanish commented on code in PR #12806:
URL: https://github.com/apache/pinot/pull/12806#discussion_r1556458423
##########
pinot-plugins/pinot-stream-ingestion/pinot-kinesis/src/main/java/org/apache/pinot/plugin/stream/kinesis/KinesisConsumer.java:
##########
@@ -69,113 +60,56 @@ public KinesisConsumer(KinesisConfig config, KinesisClient kinesisClient) {
super(config, kinesisClient);
}
- /**
- * Fetch records from the Kinesis stream between the start and end KinesisCheckpoint
- */
@Override
- public KinesisMessageBatch fetchMessages(StreamPartitionMsgOffset startMsgOffset, int timeoutMs) {
+ public synchronized KinesisMessageBatch fetchMessages(StreamPartitionMsgOffset startMsgOffset, int timeoutMs) {
KinesisPartitionGroupOffset startOffset = (KinesisPartitionGroupOffset) startMsgOffset;
- List<BytesStreamMessage> messages = new ArrayList<>();
- Future<KinesisMessageBatch> kinesisFetchResultFuture =
- _executorService.submit(() -> getResult(startOffset, messages));
- try {
- return kinesisFetchResultFuture.get(timeoutMs, TimeUnit.MILLISECONDS);
- } catch (TimeoutException e) {
- kinesisFetchResultFuture.cancel(true);
- } catch (Exception e) {
- // Ignored
- }
- return buildKinesisMessageBatch(startOffset, messages, false);
- }
-
- private KinesisMessageBatch getResult(KinesisPartitionGroupOffset startOffset, List<BytesStreamMessage> messages) {
- try {
- String shardId = startOffset.getShardId();
- String shardIterator = getShardIterator(shardId, startOffset.getSequenceNumber());
- boolean endOfShard = false;
- long currentWindow = System.currentTimeMillis() / SLEEP_TIME_BETWEEN_REQUESTS;
- int currentWindowRequests = 0;
- while (shardIterator != null) {
- GetRecordsRequest getRecordsRequest = GetRecordsRequest.builder().shardIterator(shardIterator).build();
- long requestSentTime = System.currentTimeMillis() / 1000;
- GetRecordsResponse getRecordsResponse = _kinesisClient.getRecords(getRecordsRequest);
- List<Record> records = getRecordsResponse.records();
- if (!records.isEmpty()) {
- for (Record record : records) {
- messages.add(extractStreamMessage(record, shardId));
- }
- if (messages.size() >= _config.getNumMaxRecordsToFetch()) {
- break;
- }
+ String shardId = startOffset.getShardId();
+ String startSequenceNumber = startOffset.getSequenceNumber();
+
+ // NOTE: Kinesis enforces a limit of 5 getRecords request per second on each shard from AWS end, beyond which we
+ // start getting ProvisionedThroughputExceededException. Rate limit the requests to avoid this.
+ long currentTimeMs = System.currentTimeMillis();
+ int currentTimeSeconds = (int) TimeUnit.MILLISECONDS.toSeconds(currentTimeMs);
+ if (currentTimeSeconds == _currentSecond) {
+ if (_numRequestsInCurrentSecond == _config.getRpsLimit()) {
+ try {
+ Thread.sleep(1000 - (currentTimeMs % 1000));
+ } catch (InterruptedException e) {
+ throw new RuntimeException(e);
}
-
- if (getRecordsResponse.hasChildShards() && !getRecordsResponse.childShards().isEmpty()) {
- //This statement returns true only when end of current shard has reached.
- // hasChildShards only checks if the childShard is null and is a valid instance.
- endOfShard = true;
- break;
- }
-
- shardIterator = getRecordsResponse.nextShardIterator();
-
- if (Thread.interrupted()) {
- break;
- }
-
- // Kinesis enforces a limit of 5 .getRecords request per second on each shard from AWS end
- // Beyond this limit we start getting ProvisionedThroughputExceededException which affect the ingestion
- if (requestSentTime == currentWindow) {
- currentWindowRequests++;
- } else if (requestSentTime > currentWindow) {
- currentWindow = requestSentTime;
- currentWindowRequests = 0;
- }
-
- if (currentWindowRequests >= _config.getNumMaxRecordsToFetch()) {
- try {
- Thread.sleep(SLEEP_TIME_BETWEEN_REQUESTS);
- } catch (InterruptedException e) {
- LOGGER.debug("Sleep interrupted while rate limiting Kinesis requests", e);
- break;
- }
- }
- }
-
- return buildKinesisMessageBatch(startOffset, messages, endOfShard);
- } catch (IllegalStateException e) {
- debugOrLogWarning("Illegal state exception, connection is broken", e);
- } catch (ProvisionedThroughputExceededException e) {
- debugOrLogWarning("The request rate for the stream is too high", e);
- } catch (ExpiredIteratorException e) {
- debugOrLogWarning("ShardIterator expired while trying to fetch records", e);
- } catch (ResourceNotFoundException | InvalidArgumentException e) {
- // aws errors
- LOGGER.error("Encountered AWS error while attempting to fetch records", e);
- } catch (KinesisException e) {
- debugOrLogWarning("Encountered unknown unrecoverable AWS exception", e);
- throw new RuntimeException(e);
- } catch (AbortedException e) {
- if (!(e.getCause() instanceof InterruptedException)) {
- debugOrLogWarning("Task aborted due to exception", e);
+ _currentSecond++;
+ _numRequestsInCurrentSecond = 1;
+ } else {
+ _numRequestsInCurrentSecond++;
}
- } catch (Throwable e) {
- // non transient errors
- LOGGER.error("Unknown fetchRecords exception", e);
- throw new RuntimeException(e);
+ } else {
+ _currentSecond = currentTimeSeconds;
+ _numRequestsInCurrentSecond = 1;
}
- return buildKinesisMessageBatch(startOffset, messages, false);
- }
- private void debugOrLogWarning(String message, Throwable throwable) {
- if (LOGGER.isDebugEnabled()) {
- LOGGER.debug(message, throwable);
+ // Get the shard iterator
+ String shardIterator;
+ if (startSequenceNumber.equals(_nextStartSequenceNumber)) {
+ shardIterator = _nextShardIterator;
} else {
- LOGGER.warn(message + ": " + throwable.getMessage());
+ // TODO: Revisit this logic to see if we always miss the first message when consuming from a new shard
Review Comment:
Could you add more explanation to this ? Why would we miss the 1st message.
##########
pinot-plugins/pinot-stream-ingestion/pinot-kinesis/src/main/java/org/apache/pinot/plugin/stream/kinesis/KinesisConsumer.java:
##########
@@ -69,113 +60,56 @@ public KinesisConsumer(KinesisConfig config, KinesisClient kinesisClient) {
super(config, kinesisClient);
}
- /**
- * Fetch records from the Kinesis stream between the start and end KinesisCheckpoint
- */
@Override
- public KinesisMessageBatch fetchMessages(StreamPartitionMsgOffset startMsgOffset, int timeoutMs) {
+ public synchronized KinesisMessageBatch fetchMessages(StreamPartitionMsgOffset startMsgOffset, int timeoutMs) {
KinesisPartitionGroupOffset startOffset = (KinesisPartitionGroupOffset) startMsgOffset;
- List<BytesStreamMessage> messages = new ArrayList<>();
- Future<KinesisMessageBatch> kinesisFetchResultFuture =
- _executorService.submit(() -> getResult(startOffset, messages));
- try {
- return kinesisFetchResultFuture.get(timeoutMs, TimeUnit.MILLISECONDS);
- } catch (TimeoutException e) {
- kinesisFetchResultFuture.cancel(true);
- } catch (Exception e) {
- // Ignored
- }
- return buildKinesisMessageBatch(startOffset, messages, false);
- }
-
- private KinesisMessageBatch getResult(KinesisPartitionGroupOffset startOffset, List<BytesStreamMessage> messages) {
- try {
- String shardId = startOffset.getShardId();
- String shardIterator = getShardIterator(shardId, startOffset.getSequenceNumber());
- boolean endOfShard = false;
- long currentWindow = System.currentTimeMillis() / SLEEP_TIME_BETWEEN_REQUESTS;
- int currentWindowRequests = 0;
- while (shardIterator != null) {
- GetRecordsRequest getRecordsRequest = GetRecordsRequest.builder().shardIterator(shardIterator).build();
- long requestSentTime = System.currentTimeMillis() / 1000;
- GetRecordsResponse getRecordsResponse = _kinesisClient.getRecords(getRecordsRequest);
- List<Record> records = getRecordsResponse.records();
- if (!records.isEmpty()) {
- for (Record record : records) {
- messages.add(extractStreamMessage(record, shardId));
- }
- if (messages.size() >= _config.getNumMaxRecordsToFetch()) {
- break;
- }
+ String shardId = startOffset.getShardId();
+ String startSequenceNumber = startOffset.getSequenceNumber();
+
+ // NOTE: Kinesis enforces a limit of 5 getRecords request per second on each shard from AWS end, beyond which we
+ // start getting ProvisionedThroughputExceededException. Rate limit the requests to avoid this.
+ long currentTimeMs = System.currentTimeMillis();
Review Comment:
Do we need our own custom rate limiter here ? Does kinesis client provide options to do the same thing/handle this, instead of us having this logic.
##########
pinot-plugins/pinot-stream-ingestion/pinot-kinesis/src/main/java/org/apache/pinot/plugin/stream/kinesis/KinesisConsumer.java:
##########
@@ -69,113 +60,56 @@ public KinesisConsumer(KinesisConfig config, KinesisClient kinesisClient) {
super(config, kinesisClient);
}
- /**
- * Fetch records from the Kinesis stream between the start and end KinesisCheckpoint
- */
@Override
- public KinesisMessageBatch fetchMessages(StreamPartitionMsgOffset startMsgOffset, int timeoutMs) {
+ public synchronized KinesisMessageBatch fetchMessages(StreamPartitionMsgOffset startMsgOffset, int timeoutMs) {
KinesisPartitionGroupOffset startOffset = (KinesisPartitionGroupOffset) startMsgOffset;
- List<BytesStreamMessage> messages = new ArrayList<>();
- Future<KinesisMessageBatch> kinesisFetchResultFuture =
- _executorService.submit(() -> getResult(startOffset, messages));
- try {
- return kinesisFetchResultFuture.get(timeoutMs, TimeUnit.MILLISECONDS);
- } catch (TimeoutException e) {
- kinesisFetchResultFuture.cancel(true);
- } catch (Exception e) {
- // Ignored
- }
- return buildKinesisMessageBatch(startOffset, messages, false);
- }
-
- private KinesisMessageBatch getResult(KinesisPartitionGroupOffset startOffset, List<BytesStreamMessage> messages) {
- try {
- String shardId = startOffset.getShardId();
- String shardIterator = getShardIterator(shardId, startOffset.getSequenceNumber());
- boolean endOfShard = false;
- long currentWindow = System.currentTimeMillis() / SLEEP_TIME_BETWEEN_REQUESTS;
- int currentWindowRequests = 0;
- while (shardIterator != null) {
- GetRecordsRequest getRecordsRequest = GetRecordsRequest.builder().shardIterator(shardIterator).build();
- long requestSentTime = System.currentTimeMillis() / 1000;
- GetRecordsResponse getRecordsResponse = _kinesisClient.getRecords(getRecordsRequest);
- List<Record> records = getRecordsResponse.records();
- if (!records.isEmpty()) {
- for (Record record : records) {
- messages.add(extractStreamMessage(record, shardId));
- }
- if (messages.size() >= _config.getNumMaxRecordsToFetch()) {
- break;
- }
+ String shardId = startOffset.getShardId();
+ String startSequenceNumber = startOffset.getSequenceNumber();
+
+ // NOTE: Kinesis enforces a limit of 5 getRecords request per second on each shard from AWS end, beyond which we
+ // start getting ProvisionedThroughputExceededException. Rate limit the requests to avoid this.
+ long currentTimeMs = System.currentTimeMillis();
+ int currentTimeSeconds = (int) TimeUnit.MILLISECONDS.toSeconds(currentTimeMs);
+ if (currentTimeSeconds == _currentSecond) {
+ if (_numRequestsInCurrentSecond == _config.getRpsLimit()) {
+ try {
+ Thread.sleep(1000 - (currentTimeMs % 1000));
+ } catch (InterruptedException e) {
+ throw new RuntimeException(e);
}
-
- if (getRecordsResponse.hasChildShards() && !getRecordsResponse.childShards().isEmpty()) {
- //This statement returns true only when end of current shard has reached.
- // hasChildShards only checks if the childShard is null and is a valid instance.
- endOfShard = true;
- break;
- }
-
- shardIterator = getRecordsResponse.nextShardIterator();
-
- if (Thread.interrupted()) {
- break;
- }
-
- // Kinesis enforces a limit of 5 .getRecords request per second on each shard from AWS end
- // Beyond this limit we start getting ProvisionedThroughputExceededException which affect the ingestion
- if (requestSentTime == currentWindow) {
- currentWindowRequests++;
- } else if (requestSentTime > currentWindow) {
- currentWindow = requestSentTime;
- currentWindowRequests = 0;
- }
-
- if (currentWindowRequests >= _config.getNumMaxRecordsToFetch()) {
- try {
- Thread.sleep(SLEEP_TIME_BETWEEN_REQUESTS);
- } catch (InterruptedException e) {
- LOGGER.debug("Sleep interrupted while rate limiting Kinesis requests", e);
- break;
- }
- }
- }
-
- return buildKinesisMessageBatch(startOffset, messages, endOfShard);
- } catch (IllegalStateException e) {
- debugOrLogWarning("Illegal state exception, connection is broken", e);
- } catch (ProvisionedThroughputExceededException e) {
- debugOrLogWarning("The request rate for the stream is too high", e);
- } catch (ExpiredIteratorException e) {
- debugOrLogWarning("ShardIterator expired while trying to fetch records", e);
- } catch (ResourceNotFoundException | InvalidArgumentException e) {
- // aws errors
- LOGGER.error("Encountered AWS error while attempting to fetch records", e);
- } catch (KinesisException e) {
- debugOrLogWarning("Encountered unknown unrecoverable AWS exception", e);
- throw new RuntimeException(e);
- } catch (AbortedException e) {
- if (!(e.getCause() instanceof InterruptedException)) {
- debugOrLogWarning("Task aborted due to exception", e);
+ _currentSecond++;
+ _numRequestsInCurrentSecond = 1;
+ } else {
+ _numRequestsInCurrentSecond++;
}
- } catch (Throwable e) {
- // non transient errors
- LOGGER.error("Unknown fetchRecords exception", e);
- throw new RuntimeException(e);
+ } else {
+ _currentSecond = currentTimeSeconds;
+ _numRequestsInCurrentSecond = 1;
}
- return buildKinesisMessageBatch(startOffset, messages, false);
- }
- private void debugOrLogWarning(String message, Throwable throwable) {
- if (LOGGER.isDebugEnabled()) {
- LOGGER.debug(message, throwable);
+ // Get the shard iterator
+ String shardIterator;
+ if (startSequenceNumber.equals(_nextStartSequenceNumber)) {
+ shardIterator = _nextShardIterator;
} else {
- LOGGER.warn(message + ": " + throwable.getMessage());
+ // TODO: Revisit this logic to see if we always miss the first message when consuming from a new shard
+ GetShardIteratorRequest getShardIteratorRequest =
+ GetShardIteratorRequest.builder().streamName(_config.getStreamTopicName()).shardId(shardId)
+ .startingSequenceNumber(startSequenceNumber).shardIteratorType(ShardIteratorType.AFTER_SEQUENCE_NUMBER)
+ .build();
+ shardIterator = _kinesisClient.getShardIterator(getShardIteratorRequest).shardIterator();
+ }
+ if (shardIterator == null) {
+ return new KinesisMessageBatch(List.of(), startOffset, true);
}
- }
- private KinesisMessageBatch buildKinesisMessageBatch(KinesisPartitionGroupOffset startOffset,
- List<BytesStreamMessage> messages, boolean endOfShard) {
+ // Read records
+ GetRecordsRequest getRecordRequest =
+ GetRecordsRequest.builder().shardIterator(shardIterator).limit(_config.getNumMaxRecordsToFetch()).build();
+ GetRecordsResponse getRecordsResponse = _kinesisClient.getRecords(getRecordRequest);
Review Comment:
This can be empty right, even if the stream has some data, given how kinesis works? We'll be return a response even if its empty
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Enhance Kinesis consumer [pinot]
Posted by "swaminathanmanish (via GitHub)" <gi...@apache.org>.
swaminathanmanish commented on code in PR #12806:
URL: https://github.com/apache/pinot/pull/12806#discussion_r1556650280
##########
pinot-plugins/pinot-stream-ingestion/pinot-kinesis/src/main/java/org/apache/pinot/plugin/stream/kinesis/KinesisConsumer.java:
##########
@@ -69,113 +60,56 @@ public KinesisConsumer(KinesisConfig config, KinesisClient kinesisClient) {
super(config, kinesisClient);
}
- /**
- * Fetch records from the Kinesis stream between the start and end KinesisCheckpoint
- */
@Override
- public KinesisMessageBatch fetchMessages(StreamPartitionMsgOffset startMsgOffset, int timeoutMs) {
+ public synchronized KinesisMessageBatch fetchMessages(StreamPartitionMsgOffset startMsgOffset, int timeoutMs) {
KinesisPartitionGroupOffset startOffset = (KinesisPartitionGroupOffset) startMsgOffset;
- List<BytesStreamMessage> messages = new ArrayList<>();
- Future<KinesisMessageBatch> kinesisFetchResultFuture =
- _executorService.submit(() -> getResult(startOffset, messages));
- try {
- return kinesisFetchResultFuture.get(timeoutMs, TimeUnit.MILLISECONDS);
- } catch (TimeoutException e) {
- kinesisFetchResultFuture.cancel(true);
- } catch (Exception e) {
- // Ignored
- }
- return buildKinesisMessageBatch(startOffset, messages, false);
- }
-
- private KinesisMessageBatch getResult(KinesisPartitionGroupOffset startOffset, List<BytesStreamMessage> messages) {
- try {
- String shardId = startOffset.getShardId();
- String shardIterator = getShardIterator(shardId, startOffset.getSequenceNumber());
- boolean endOfShard = false;
- long currentWindow = System.currentTimeMillis() / SLEEP_TIME_BETWEEN_REQUESTS;
- int currentWindowRequests = 0;
- while (shardIterator != null) {
- GetRecordsRequest getRecordsRequest = GetRecordsRequest.builder().shardIterator(shardIterator).build();
- long requestSentTime = System.currentTimeMillis() / 1000;
- GetRecordsResponse getRecordsResponse = _kinesisClient.getRecords(getRecordsRequest);
- List<Record> records = getRecordsResponse.records();
- if (!records.isEmpty()) {
- for (Record record : records) {
- messages.add(extractStreamMessage(record, shardId));
- }
- if (messages.size() >= _config.getNumMaxRecordsToFetch()) {
- break;
- }
+ String shardId = startOffset.getShardId();
+ String startSequenceNumber = startOffset.getSequenceNumber();
+
+ // NOTE: Kinesis enforces a limit of 5 getRecords request per second on each shard from AWS end, beyond which we
+ // start getting ProvisionedThroughputExceededException. Rate limit the requests to avoid this.
+ long currentTimeMs = System.currentTimeMillis();
+ int currentTimeSeconds = (int) TimeUnit.MILLISECONDS.toSeconds(currentTimeMs);
+ if (currentTimeSeconds == _currentSecond) {
+ if (_numRequestsInCurrentSecond == _config.getRpsLimit()) {
+ try {
+ Thread.sleep(1000 - (currentTimeMs % 1000));
+ } catch (InterruptedException e) {
+ throw new RuntimeException(e);
}
-
- if (getRecordsResponse.hasChildShards() && !getRecordsResponse.childShards().isEmpty()) {
- //This statement returns true only when end of current shard has reached.
- // hasChildShards only checks if the childShard is null and is a valid instance.
- endOfShard = true;
- break;
- }
-
- shardIterator = getRecordsResponse.nextShardIterator();
-
- if (Thread.interrupted()) {
- break;
- }
-
- // Kinesis enforces a limit of 5 .getRecords request per second on each shard from AWS end
- // Beyond this limit we start getting ProvisionedThroughputExceededException which affect the ingestion
- if (requestSentTime == currentWindow) {
- currentWindowRequests++;
- } else if (requestSentTime > currentWindow) {
- currentWindow = requestSentTime;
- currentWindowRequests = 0;
- }
-
- if (currentWindowRequests >= _config.getNumMaxRecordsToFetch()) {
- try {
- Thread.sleep(SLEEP_TIME_BETWEEN_REQUESTS);
- } catch (InterruptedException e) {
- LOGGER.debug("Sleep interrupted while rate limiting Kinesis requests", e);
- break;
- }
- }
- }
-
- return buildKinesisMessageBatch(startOffset, messages, endOfShard);
- } catch (IllegalStateException e) {
- debugOrLogWarning("Illegal state exception, connection is broken", e);
- } catch (ProvisionedThroughputExceededException e) {
- debugOrLogWarning("The request rate for the stream is too high", e);
- } catch (ExpiredIteratorException e) {
- debugOrLogWarning("ShardIterator expired while trying to fetch records", e);
- } catch (ResourceNotFoundException | InvalidArgumentException e) {
- // aws errors
- LOGGER.error("Encountered AWS error while attempting to fetch records", e);
- } catch (KinesisException e) {
- debugOrLogWarning("Encountered unknown unrecoverable AWS exception", e);
- throw new RuntimeException(e);
- } catch (AbortedException e) {
- if (!(e.getCause() instanceof InterruptedException)) {
- debugOrLogWarning("Task aborted due to exception", e);
+ _currentSecond++;
+ _numRequestsInCurrentSecond = 1;
+ } else {
+ _numRequestsInCurrentSecond++;
}
- } catch (Throwable e) {
- // non transient errors
- LOGGER.error("Unknown fetchRecords exception", e);
- throw new RuntimeException(e);
+ } else {
+ _currentSecond = currentTimeSeconds;
+ _numRequestsInCurrentSecond = 1;
}
- return buildKinesisMessageBatch(startOffset, messages, false);
- }
- private void debugOrLogWarning(String message, Throwable throwable) {
- if (LOGGER.isDebugEnabled()) {
- LOGGER.debug(message, throwable);
+ // Get the shard iterator
+ String shardIterator;
+ if (startSequenceNumber.equals(_nextStartSequenceNumber)) {
+ shardIterator = _nextShardIterator;
} else {
- LOGGER.warn(message + ": " + throwable.getMessage());
+ // TODO: Revisit this logic to see if we always miss the first message when consuming from a new shard
+ GetShardIteratorRequest getShardIteratorRequest =
+ GetShardIteratorRequest.builder().streamName(_config.getStreamTopicName()).shardId(shardId)
+ .startingSequenceNumber(startSequenceNumber).shardIteratorType(ShardIteratorType.AFTER_SEQUENCE_NUMBER)
+ .build();
+ shardIterator = _kinesisClient.getShardIterator(getShardIteratorRequest).shardIterator();
+ }
+ if (shardIterator == null) {
+ return new KinesisMessageBatch(List.of(), startOffset, true);
}
- }
- private KinesisMessageBatch buildKinesisMessageBatch(KinesisPartitionGroupOffset startOffset,
- List<BytesStreamMessage> messages, boolean endOfShard) {
+ // Read records
+ GetRecordsRequest getRecordRequest =
+ GetRecordsRequest.builder().shardIterator(shardIterator).limit(_config.getNumMaxRecordsToFetch()).build();
+ GetRecordsResponse getRecordsResponse = _kinesisClient.getRecords(getRecordRequest);
Review Comment:
Good point. We can also a metric to track this when it happens.
##########
pinot-plugins/pinot-stream-ingestion/pinot-kinesis/src/main/java/org/apache/pinot/plugin/stream/kinesis/KinesisConsumer.java:
##########
@@ -69,134 +61,77 @@ public KinesisConsumer(KinesisConfig config, KinesisClient kinesisClient) {
super(config, kinesisClient);
}
- /**
- * Fetch records from the Kinesis stream between the start and end KinesisCheckpoint
- */
@Override
- public KinesisMessageBatch fetchMessages(StreamPartitionMsgOffset startMsgOffset, int timeoutMs) {
+ public synchronized KinesisMessageBatch fetchMessages(StreamPartitionMsgOffset startMsgOffset, int timeoutMs) {
KinesisPartitionGroupOffset startOffset = (KinesisPartitionGroupOffset) startMsgOffset;
- List<BytesStreamMessage> messages = new ArrayList<>();
- Future<KinesisMessageBatch> kinesisFetchResultFuture =
- _executorService.submit(() -> getResult(startOffset, messages));
- try {
- return kinesisFetchResultFuture.get(timeoutMs, TimeUnit.MILLISECONDS);
- } catch (TimeoutException e) {
- kinesisFetchResultFuture.cancel(true);
- } catch (Exception e) {
- // Ignored
- }
- return buildKinesisMessageBatch(startOffset, messages, false);
- }
-
- private KinesisMessageBatch getResult(KinesisPartitionGroupOffset startOffset, List<BytesStreamMessage> messages) {
- try {
- String shardId = startOffset.getShardId();
- String shardIterator = getShardIterator(shardId, startOffset.getSequenceNumber());
- boolean endOfShard = false;
- long currentWindow = System.currentTimeMillis() / SLEEP_TIME_BETWEEN_REQUESTS;
- int currentWindowRequests = 0;
- while (shardIterator != null) {
- GetRecordsRequest getRecordsRequest = GetRecordsRequest.builder().shardIterator(shardIterator).build();
- long requestSentTime = System.currentTimeMillis() / 1000;
- GetRecordsResponse getRecordsResponse = _kinesisClient.getRecords(getRecordsRequest);
- List<Record> records = getRecordsResponse.records();
- if (!records.isEmpty()) {
- for (Record record : records) {
- messages.add(extractStreamMessage(record, shardId));
- }
- if (messages.size() >= _config.getNumMaxRecordsToFetch()) {
- break;
- }
- }
-
- if (getRecordsResponse.hasChildShards() && !getRecordsResponse.childShards().isEmpty()) {
- //This statement returns true only when end of current shard has reached.
- // hasChildShards only checks if the childShard is null and is a valid instance.
- endOfShard = true;
- break;
- }
-
- shardIterator = getRecordsResponse.nextShardIterator();
-
- if (Thread.interrupted()) {
- break;
- }
-
- // Kinesis enforces a limit of 5 .getRecords request per second on each shard from AWS end
- // Beyond this limit we start getting ProvisionedThroughputExceededException which affect the ingestion
- if (requestSentTime == currentWindow) {
- currentWindowRequests++;
- } else if (requestSentTime > currentWindow) {
- currentWindow = requestSentTime;
- currentWindowRequests = 0;
- }
-
- if (currentWindowRequests >= _config.getNumMaxRecordsToFetch()) {
- try {
- Thread.sleep(SLEEP_TIME_BETWEEN_REQUESTS);
- } catch (InterruptedException e) {
- LOGGER.debug("Sleep interrupted while rate limiting Kinesis requests", e);
- break;
- }
- }
- }
-
- return buildKinesisMessageBatch(startOffset, messages, endOfShard);
- } catch (IllegalStateException e) {
- debugOrLogWarning("Illegal state exception, connection is broken", e);
- } catch (ProvisionedThroughputExceededException e) {
- debugOrLogWarning("The request rate for the stream is too high", e);
- } catch (ExpiredIteratorException e) {
- debugOrLogWarning("ShardIterator expired while trying to fetch records", e);
- } catch (ResourceNotFoundException | InvalidArgumentException e) {
- // aws errors
- LOGGER.error("Encountered AWS error while attempting to fetch records", e);
- } catch (KinesisException e) {
- debugOrLogWarning("Encountered unknown unrecoverable AWS exception", e);
- throw new RuntimeException(e);
- } catch (AbortedException e) {
- if (!(e.getCause() instanceof InterruptedException)) {
- debugOrLogWarning("Task aborted due to exception", e);
- }
- } catch (Throwable e) {
- // non transient errors
- LOGGER.error("Unknown fetchRecords exception", e);
- throw new RuntimeException(e);
- }
- return buildKinesisMessageBatch(startOffset, messages, false);
- }
+ String shardId = startOffset.getShardId();
+ String startSequenceNumber = startOffset.getSequenceNumber();
- private void debugOrLogWarning(String message, Throwable throwable) {
- if (LOGGER.isDebugEnabled()) {
- LOGGER.debug(message, throwable);
+ // Get the shard iterator
+ String shardIterator;
+ if (startSequenceNumber.equals(_nextStartSequenceNumber)) {
+ shardIterator = _nextShardIterator;
} else {
- LOGGER.warn(message + ": " + throwable.getMessage());
+ // TODO: Revisit the offset handling logic. Reading after the start sequence number can lose the first message
+ // when consuming from a new partition because the initial start sequence number is inclusive.
+ GetShardIteratorRequest getShardIteratorRequest =
+ GetShardIteratorRequest.builder().streamName(_config.getStreamTopicName()).shardId(shardId)
+ .startingSequenceNumber(startSequenceNumber).shardIteratorType(ShardIteratorType.AFTER_SEQUENCE_NUMBER)
+ .build();
+ shardIterator = _kinesisClient.getShardIterator(getShardIteratorRequest).shardIterator();
+ }
+ if (shardIterator == null) {
+ return new KinesisMessageBatch(List.of(), startOffset, true);
}
- }
- private KinesisMessageBatch buildKinesisMessageBatch(KinesisPartitionGroupOffset startOffset,
- List<BytesStreamMessage> messages, boolean endOfShard) {
+ // Read records
+ rateLimitRequests();
+ GetRecordsRequest getRecordRequest =
+ GetRecordsRequest.builder().shardIterator(shardIterator).limit(_config.getNumMaxRecordsToFetch()).build();
+ GetRecordsResponse getRecordsResponse = _kinesisClient.getRecords(getRecordRequest);
+ List<Record> records = getRecordsResponse.records();
+ List<BytesStreamMessage> messages;
KinesisPartitionGroupOffset offsetOfNextBatch;
- if (messages.isEmpty()) {
- offsetOfNextBatch = startOffset;
- } else {
+ if (!records.isEmpty()) {
+ messages = records.stream().map(record -> extractStreamMessage(record, shardId)).collect(Collectors.toList());
StreamMessageMetadata lastMessageMetadata = messages.get(messages.size() - 1).getMetadata();
assert lastMessageMetadata != null;
offsetOfNextBatch = (KinesisPartitionGroupOffset) lastMessageMetadata.getNextOffset();
+ } else {
+ // TODO: Revisit whether Kinesis can return empty batch when there are available records. The consumer cna handle
+ // empty message batch, but it will treat it as fully caught up.
+ messages = List.of();
+ offsetOfNextBatch = startOffset;
}
- return new KinesisMessageBatch(messages, offsetOfNextBatch, endOfShard);
+ assert offsetOfNextBatch != null;
+ _nextStartSequenceNumber = offsetOfNextBatch.getSequenceNumber();
+ _nextShardIterator = getRecordsResponse.nextShardIterator();
+ return new KinesisMessageBatch(messages, offsetOfNextBatch, _nextShardIterator == null);
}
- private String getShardIterator(String shardId, String sequenceNumber) {
- GetShardIteratorRequest.Builder requestBuilder =
- GetShardIteratorRequest.builder().streamName(_config.getStreamTopicName()).shardId(shardId);
- if (sequenceNumber != null) {
- requestBuilder = requestBuilder.startingSequenceNumber(sequenceNumber)
- .shardIteratorType(ShardIteratorType.AFTER_SEQUENCE_NUMBER);
+ /**
+ * Kinesis enforces a limit of 5 getRecords request per second on each shard from AWS end, beyond which we start
+ * getting {@link ProvisionedThroughputExceededException}. Rate limit the requests to avoid this.
+ */
+ private void rateLimitRequests() {
Review Comment:
Thanks for creating a separate method. I guess this being a special kind of rate limiter that needs to block until we are ready to fetch again, we cannot leverage off the shelf ones like guava.
if kinesis has a limit, don't we need to adhere to that limit. So does getRpsLimit() need to be what Kinesis limit is ?
##########
pinot-plugins/pinot-stream-ingestion/pinot-kinesis/src/main/java/org/apache/pinot/plugin/stream/kinesis/KinesisConsumer.java:
##########
@@ -69,134 +61,77 @@ public KinesisConsumer(KinesisConfig config, KinesisClient kinesisClient) {
super(config, kinesisClient);
}
- /**
- * Fetch records from the Kinesis stream between the start and end KinesisCheckpoint
- */
@Override
- public KinesisMessageBatch fetchMessages(StreamPartitionMsgOffset startMsgOffset, int timeoutMs) {
+ public synchronized KinesisMessageBatch fetchMessages(StreamPartitionMsgOffset startMsgOffset, int timeoutMs) {
KinesisPartitionGroupOffset startOffset = (KinesisPartitionGroupOffset) startMsgOffset;
- List<BytesStreamMessage> messages = new ArrayList<>();
- Future<KinesisMessageBatch> kinesisFetchResultFuture =
- _executorService.submit(() -> getResult(startOffset, messages));
- try {
- return kinesisFetchResultFuture.get(timeoutMs, TimeUnit.MILLISECONDS);
- } catch (TimeoutException e) {
- kinesisFetchResultFuture.cancel(true);
- } catch (Exception e) {
- // Ignored
- }
- return buildKinesisMessageBatch(startOffset, messages, false);
- }
-
- private KinesisMessageBatch getResult(KinesisPartitionGroupOffset startOffset, List<BytesStreamMessage> messages) {
- try {
- String shardId = startOffset.getShardId();
- String shardIterator = getShardIterator(shardId, startOffset.getSequenceNumber());
- boolean endOfShard = false;
- long currentWindow = System.currentTimeMillis() / SLEEP_TIME_BETWEEN_REQUESTS;
- int currentWindowRequests = 0;
- while (shardIterator != null) {
- GetRecordsRequest getRecordsRequest = GetRecordsRequest.builder().shardIterator(shardIterator).build();
- long requestSentTime = System.currentTimeMillis() / 1000;
- GetRecordsResponse getRecordsResponse = _kinesisClient.getRecords(getRecordsRequest);
- List<Record> records = getRecordsResponse.records();
- if (!records.isEmpty()) {
- for (Record record : records) {
- messages.add(extractStreamMessage(record, shardId));
- }
- if (messages.size() >= _config.getNumMaxRecordsToFetch()) {
- break;
- }
- }
-
- if (getRecordsResponse.hasChildShards() && !getRecordsResponse.childShards().isEmpty()) {
- //This statement returns true only when end of current shard has reached.
- // hasChildShards only checks if the childShard is null and is a valid instance.
- endOfShard = true;
- break;
- }
-
- shardIterator = getRecordsResponse.nextShardIterator();
-
- if (Thread.interrupted()) {
- break;
- }
-
- // Kinesis enforces a limit of 5 .getRecords request per second on each shard from AWS end
- // Beyond this limit we start getting ProvisionedThroughputExceededException which affect the ingestion
- if (requestSentTime == currentWindow) {
- currentWindowRequests++;
- } else if (requestSentTime > currentWindow) {
- currentWindow = requestSentTime;
- currentWindowRequests = 0;
- }
-
- if (currentWindowRequests >= _config.getNumMaxRecordsToFetch()) {
- try {
- Thread.sleep(SLEEP_TIME_BETWEEN_REQUESTS);
- } catch (InterruptedException e) {
- LOGGER.debug("Sleep interrupted while rate limiting Kinesis requests", e);
- break;
- }
- }
- }
-
- return buildKinesisMessageBatch(startOffset, messages, endOfShard);
- } catch (IllegalStateException e) {
- debugOrLogWarning("Illegal state exception, connection is broken", e);
- } catch (ProvisionedThroughputExceededException e) {
- debugOrLogWarning("The request rate for the stream is too high", e);
- } catch (ExpiredIteratorException e) {
- debugOrLogWarning("ShardIterator expired while trying to fetch records", e);
- } catch (ResourceNotFoundException | InvalidArgumentException e) {
- // aws errors
- LOGGER.error("Encountered AWS error while attempting to fetch records", e);
- } catch (KinesisException e) {
- debugOrLogWarning("Encountered unknown unrecoverable AWS exception", e);
- throw new RuntimeException(e);
- } catch (AbortedException e) {
- if (!(e.getCause() instanceof InterruptedException)) {
- debugOrLogWarning("Task aborted due to exception", e);
- }
- } catch (Throwable e) {
- // non transient errors
- LOGGER.error("Unknown fetchRecords exception", e);
- throw new RuntimeException(e);
- }
- return buildKinesisMessageBatch(startOffset, messages, false);
- }
+ String shardId = startOffset.getShardId();
+ String startSequenceNumber = startOffset.getSequenceNumber();
- private void debugOrLogWarning(String message, Throwable throwable) {
- if (LOGGER.isDebugEnabled()) {
- LOGGER.debug(message, throwable);
+ // Get the shard iterator
+ String shardIterator;
+ if (startSequenceNumber.equals(_nextStartSequenceNumber)) {
+ shardIterator = _nextShardIterator;
} else {
- LOGGER.warn(message + ": " + throwable.getMessage());
+ // TODO: Revisit the offset handling logic. Reading after the start sequence number can lose the first message
+ // when consuming from a new partition because the initial start sequence number is inclusive.
+ GetShardIteratorRequest getShardIteratorRequest =
+ GetShardIteratorRequest.builder().streamName(_config.getStreamTopicName()).shardId(shardId)
+ .startingSequenceNumber(startSequenceNumber).shardIteratorType(ShardIteratorType.AFTER_SEQUENCE_NUMBER)
+ .build();
+ shardIterator = _kinesisClient.getShardIterator(getShardIteratorRequest).shardIterator();
+ }
+ if (shardIterator == null) {
+ return new KinesisMessageBatch(List.of(), startOffset, true);
}
- }
- private KinesisMessageBatch buildKinesisMessageBatch(KinesisPartitionGroupOffset startOffset,
- List<BytesStreamMessage> messages, boolean endOfShard) {
+ // Read records
+ rateLimitRequests();
+ GetRecordsRequest getRecordRequest =
+ GetRecordsRequest.builder().shardIterator(shardIterator).limit(_config.getNumMaxRecordsToFetch()).build();
+ GetRecordsResponse getRecordsResponse = _kinesisClient.getRecords(getRecordRequest);
+ List<Record> records = getRecordsResponse.records();
+ List<BytesStreamMessage> messages;
KinesisPartitionGroupOffset offsetOfNextBatch;
- if (messages.isEmpty()) {
- offsetOfNextBatch = startOffset;
- } else {
+ if (!records.isEmpty()) {
+ messages = records.stream().map(record -> extractStreamMessage(record, shardId)).collect(Collectors.toList());
StreamMessageMetadata lastMessageMetadata = messages.get(messages.size() - 1).getMetadata();
assert lastMessageMetadata != null;
offsetOfNextBatch = (KinesisPartitionGroupOffset) lastMessageMetadata.getNextOffset();
+ } else {
+ // TODO: Revisit whether Kinesis can return empty batch when there are available records. The consumer cna handle
+ // empty message batch, but it will treat it as fully caught up.
+ messages = List.of();
+ offsetOfNextBatch = startOffset;
}
- return new KinesisMessageBatch(messages, offsetOfNextBatch, endOfShard);
+ assert offsetOfNextBatch != null;
+ _nextStartSequenceNumber = offsetOfNextBatch.getSequenceNumber();
+ _nextShardIterator = getRecordsResponse.nextShardIterator();
+ return new KinesisMessageBatch(messages, offsetOfNextBatch, _nextShardIterator == null);
}
- private String getShardIterator(String shardId, String sequenceNumber) {
- GetShardIteratorRequest.Builder requestBuilder =
- GetShardIteratorRequest.builder().streamName(_config.getStreamTopicName()).shardId(shardId);
- if (sequenceNumber != null) {
- requestBuilder = requestBuilder.startingSequenceNumber(sequenceNumber)
- .shardIteratorType(ShardIteratorType.AFTER_SEQUENCE_NUMBER);
+ /**
+ * Kinesis enforces a limit of 5 getRecords request per second on each shard from AWS end, beyond which we start
+ * getting {@link ProvisionedThroughputExceededException}. Rate limit the requests to avoid this.
+ */
+ private void rateLimitRequests() {
+ long currentTimeMs = System.currentTimeMillis();
+ int currentTimeSeconds = (int) TimeUnit.MILLISECONDS.toSeconds(currentTimeMs);
+ if (currentTimeSeconds == _currentSecond) {
+ if (_numRequestsInCurrentSecond == _config.getRpsLimit()) {
Review Comment:
This can be done later. A log.info or metric would help debug if rate limiting becomes an issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Enhance Kinesis consumer [pinot]
Posted by "KKcorps (via GitHub)" <gi...@apache.org>.
KKcorps commented on code in PR #12806:
URL: https://github.com/apache/pinot/pull/12806#discussion_r1584605569
##########
pinot-plugins/pinot-stream-ingestion/pinot-kinesis/src/main/java/org/apache/pinot/plugin/stream/kinesis/KinesisConsumer.java:
##########
@@ -69,134 +61,77 @@ public KinesisConsumer(KinesisConfig config, KinesisClient kinesisClient) {
super(config, kinesisClient);
}
- /**
- * Fetch records from the Kinesis stream between the start and end KinesisCheckpoint
- */
@Override
- public KinesisMessageBatch fetchMessages(StreamPartitionMsgOffset startMsgOffset, int timeoutMs) {
+ public synchronized KinesisMessageBatch fetchMessages(StreamPartitionMsgOffset startMsgOffset, int timeoutMs) {
KinesisPartitionGroupOffset startOffset = (KinesisPartitionGroupOffset) startMsgOffset;
- List<BytesStreamMessage> messages = new ArrayList<>();
- Future<KinesisMessageBatch> kinesisFetchResultFuture =
- _executorService.submit(() -> getResult(startOffset, messages));
- try {
- return kinesisFetchResultFuture.get(timeoutMs, TimeUnit.MILLISECONDS);
- } catch (TimeoutException e) {
- kinesisFetchResultFuture.cancel(true);
- } catch (Exception e) {
- // Ignored
- }
- return buildKinesisMessageBatch(startOffset, messages, false);
- }
-
- private KinesisMessageBatch getResult(KinesisPartitionGroupOffset startOffset, List<BytesStreamMessage> messages) {
- try {
- String shardId = startOffset.getShardId();
- String shardIterator = getShardIterator(shardId, startOffset.getSequenceNumber());
- boolean endOfShard = false;
- long currentWindow = System.currentTimeMillis() / SLEEP_TIME_BETWEEN_REQUESTS;
- int currentWindowRequests = 0;
- while (shardIterator != null) {
- GetRecordsRequest getRecordsRequest = GetRecordsRequest.builder().shardIterator(shardIterator).build();
- long requestSentTime = System.currentTimeMillis() / 1000;
- GetRecordsResponse getRecordsResponse = _kinesisClient.getRecords(getRecordsRequest);
- List<Record> records = getRecordsResponse.records();
- if (!records.isEmpty()) {
- for (Record record : records) {
- messages.add(extractStreamMessage(record, shardId));
- }
- if (messages.size() >= _config.getNumMaxRecordsToFetch()) {
- break;
- }
- }
-
- if (getRecordsResponse.hasChildShards() && !getRecordsResponse.childShards().isEmpty()) {
- //This statement returns true only when end of current shard has reached.
- // hasChildShards only checks if the childShard is null and is a valid instance.
- endOfShard = true;
- break;
- }
-
- shardIterator = getRecordsResponse.nextShardIterator();
-
- if (Thread.interrupted()) {
- break;
- }
-
- // Kinesis enforces a limit of 5 .getRecords request per second on each shard from AWS end
- // Beyond this limit we start getting ProvisionedThroughputExceededException which affect the ingestion
- if (requestSentTime == currentWindow) {
- currentWindowRequests++;
- } else if (requestSentTime > currentWindow) {
- currentWindow = requestSentTime;
- currentWindowRequests = 0;
- }
-
- if (currentWindowRequests >= _config.getNumMaxRecordsToFetch()) {
- try {
- Thread.sleep(SLEEP_TIME_BETWEEN_REQUESTS);
- } catch (InterruptedException e) {
- LOGGER.debug("Sleep interrupted while rate limiting Kinesis requests", e);
- break;
- }
- }
- }
-
- return buildKinesisMessageBatch(startOffset, messages, endOfShard);
- } catch (IllegalStateException e) {
- debugOrLogWarning("Illegal state exception, connection is broken", e);
- } catch (ProvisionedThroughputExceededException e) {
- debugOrLogWarning("The request rate for the stream is too high", e);
- } catch (ExpiredIteratorException e) {
- debugOrLogWarning("ShardIterator expired while trying to fetch records", e);
- } catch (ResourceNotFoundException | InvalidArgumentException e) {
- // aws errors
- LOGGER.error("Encountered AWS error while attempting to fetch records", e);
- } catch (KinesisException e) {
- debugOrLogWarning("Encountered unknown unrecoverable AWS exception", e);
- throw new RuntimeException(e);
- } catch (AbortedException e) {
- if (!(e.getCause() instanceof InterruptedException)) {
- debugOrLogWarning("Task aborted due to exception", e);
- }
- } catch (Throwable e) {
- // non transient errors
- LOGGER.error("Unknown fetchRecords exception", e);
- throw new RuntimeException(e);
- }
- return buildKinesisMessageBatch(startOffset, messages, false);
- }
+ String shardId = startOffset.getShardId();
+ String startSequenceNumber = startOffset.getSequenceNumber();
- private void debugOrLogWarning(String message, Throwable throwable) {
- if (LOGGER.isDebugEnabled()) {
- LOGGER.debug(message, throwable);
+ // Get the shard iterator
+ String shardIterator;
+ if (startSequenceNumber.equals(_nextStartSequenceNumber)) {
+ shardIterator = _nextShardIterator;
Review Comment:
will need to handle a case here when nextShardIterator has expired (since it has time limit of 5 minutes).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org