You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@camel.apache.org by GitBox <gi...@apache.org> on 2020/11/24 10:42:37 UTC
[GitHub] [camel-quarkus] springfan opened a new issue #2023: aws kinesis: De-aggregate producer side aggregated msgs
springfan opened a new issue #2023:
URL: https://github.com/apache/camel-quarkus/issues/2023
```
<quarkus-plugin.version>1.8.1.Final</quarkus-plugin.version>
<dependency>
<groupId>org.apache.camel.quarkus</groupId>
<artifactId>camel-quarkus-aws-kinesis</artifactId>
</dependency>
```
Our producer sends zipped messages. When reading those about 1/4 of all msgs cannot be unzipped. When digging into the message blob it became clear that one message contains multiple data blobs. That's probably due to the producer side message aggregation ( https://docs.aws.amazon.com/streams/latest/dev/kinesis-kpl-consumer-deaggregation.html).
Instead of a single message the data from the Record entity contains obviously several merged data blobs that I'm unable to read (zip error when trying to decompress). I'd expect to receive several msgs in my processor instead of several msgs merged into one msg blob.
```
from("aws-kinesis://" lsc.getStreamName() "?amazonKinesisClient=client&bridgeErrorHandler=true&maxResultsPerRequest=500&greedy=true&delay=0&runLoggingLevel=TRACE&iteratorType=LATEST")
.routeId("vssKinesisStream")
.log("LSC message received.")
.onException(java.util.zip.ZipException.class).process(exchange ->
{ metrics.countInvalidPerSec(); }
)
.stop()
.end()
.process(exchange ->
{ metrics.countReceivedPerSec(); Record record = exchange.getIn().getBody(Record.class); String message = extractMessageFromRecord(record); exchange.getIn().setBody(message); metrics.countValidPerSec(); LOG.info("Extracted message: " + message); lastReadMsgAge = Duration.between(record.getApproximateArrivalTimestamp().toInstant(), Instant.now()); }
```
According to https://issues.apache.org/jira/browse/CAMEL-15885 that is expected behavior. But how should one de-aggregate the merged msg blob? Isn't that a missing feature when the KCL supports automatic de-aggregation of KPL user records.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-quarkus] ppalaga closed issue #2023: aws kinesis: De-aggregate producer side aggregated msgs
Posted by GitBox <gi...@apache.org>.
ppalaga closed issue #2023:
URL: https://github.com/apache/camel-quarkus/issues/2023
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-quarkus] ppalaga commented on issue #2023: aws kinesis: De-aggregate producer side aggregated msgs
Posted by GitBox <gi...@apache.org>.
ppalaga commented on issue #2023:
URL: https://github.com/apache/camel-quarkus/issues/2023#issuecomment-848983447
AWS SDK v1 extensions were removed some time ago. If the problem occurs also with the respective v2 extension, please feel free to reopen this issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org