You are viewing a plain text version of this content. The canonical link for it is here.

Posted to github@beam.apache.org by "scwhittle (via GitHub)" <gi...@apache.org> on 2024/08/12 11:09:31 UTC

Re: [PR] Remove expensive shuffle of read data in KafkaIO when using sdf and commit offsets [beam]

scwhittle commented on PR #31682:
URL: https://github.com/apache/beam/pull/31682#issuecomment-2283680669

   This change causes an error when updating the Dataflow pipeline from previous version, but the update can be allowed by passing:
   ```
   --transformNameMapping={"KafkaIO.Read/KafkaIO.Read/KafkaIO.Read.ReadFromKafkaViaSDF/KafkaIO.ReadSourceDescriptors/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey":""}"
   ```
   
   @kennknowles Is that sufficient or should I add some option to maintain the previous expensive behavior? Are there any other concerns with this change? Thanks!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org