[GitHub] [beam] mosche edited a comment on pull request #16077: [BEAM-13175] Add KinesisIO.write for AWS SDK v2.

[GitBox <gi...@apache.org>]

mosche edited a comment on pull request #16077:
URL: https://github.com/apache/beam/pull/16077#issuecomment-1000260579


   > Also, did you have a chance to run it against a real Kinesis instance and maybe compare the performance between this version and KPL's one (for AWS SDK v1)?
   
   I did quite some runs against real Kinesis using the integration test (using `--useLocalstack=false`). But I'll try and spend some time comparing performance. 
   
   Though, the key issue will be the partitioner here. I'm not sure how to make it a meaningful & fair comparison. For instance, when using unique ids as partition key the new writer won't aggregate anything (except for some collisions on the effective hash key). On the other hand, using the provided random partitioner with the correct number of shards, it will aggregate as much as possible. The new writer certainly pushes a bit more responsibility to the user to think about partitioning upfront :/


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org