You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2022/06/04 20:51:00 UTC
[GitHub] [beam] damccorm opened a new issue, #20990: Processing message in sequence in a pipeline from IBM MQ (deliver message in order)
damccorm opened a new issue, #20990:
URL: https://github.com/apache/beam/issues/20990
We want to create a data pipeline that will read message from IBM MQ. And process the messages in the order it receive the message from IBM MQ. IBM MQ will deliver the message in order. The message does not have any intelligence (for example a timestamp or sequence number)..Would appreciate if it possible to achieve it without using single threaded model.
We want to scale it we would like to have ordering & parallelism, not sure whether its doable or not. Request your help. The runner is google dataflow.
Imported from Jira [BEAM-12499](https://issues.apache.org/jira/browse/BEAM-12499). Original Jira may contain additional context.
Reported by: ashwinidas.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kennknowles commented on issue #20990: Processing message in sequence in a pipeline from IBM MQ (deliver message in order)
Posted by GitBox <gi...@apache.org>.
kennknowles commented on issue #20990:
URL: https://github.com/apache/beam/issues/20990#issuecomment-1334618866
Order and parallelism are logically opposite. By definition things that are in parallel are not ordered, and vice versa. You can achieve parallelism with per-shard order, which is what people tend to do.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kennknowles commented on issue #20990: Processing message in sequence in a pipeline from IBM MQ (deliver message in order)
Posted by GitBox <gi...@apache.org>.
kennknowles commented on issue #20990:
URL: https://github.com/apache/beam/issues/20990#issuecomment-1334619090
I don't know anything about IBM MQ but it sounds like this is a feature request for an IO
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org