You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2023/01/11 13:09:56 UTC
[GitHub] [beam] Amraneze opened a new issue, #24971: [Bug]: Messages are not published when a connection is closed with JmsIO
Amraneze opened a new issue, #24971:
URL: https://github.com/apache/beam/issues/24971
### What happened?
When using `JmsIO` for publishing messages, we found out that messages are not being published when the connection is closed due to any reason (Network, Broker down ...). Even though we have the option to get failed published messages we don't have a way to have a retry policy.
````java
var failedPublishedMessages = messages.apply(getWriter(sinkOptions)).getFailedMessages();
failedPublishedMessages.apply(REPUBLISH_FAILED_MESSAGE_NAME, getWriter(sinkOptions));
return PDone.in(cdpMessages.getPipeline());
````
>Note: the function `getWriter` returns JMS writer `JmsIO.Write<Message>`
### Issue Priority
Priority: 1 (data loss / total loss of function)
### Issue Components
- [ ] Component: Python SDK
- [ ] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [X] Component: IO connector
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Samza Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [ ] Component: Google Cloud Dataflow Runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kennknowles commented on issue #24971: [Bug]: Messages are not published when a connection is closed with JmsIO
Posted by "kennknowles (via GitHub)" <gi...@apache.org>.
kennknowles commented on issue #24971:
URL: https://github.com/apache/beam/issues/24971#issuecomment-1426489271
Does it cause a failure of the processing of the element? In that case the runner should retry the processing with a fresh instance of the DoFn, which should retry the publish. It isn't as good as a retry policy, but it avoids data loss.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Abacn closed issue #24971: [Bug]: Messages are not published when a connection is closed with JmsIO
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn closed issue #24971: [Bug]: Messages are not published when a connection is closed with JmsIO
URL: https://github.com/apache/beam/issues/24971
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Amraneze commented on issue #24971: [Bug]: Messages are not published when a connection is closed with JmsIO
Posted by "Amraneze (via GitHub)" <gi...@apache.org>.
Amraneze commented on issue #24971:
URL: https://github.com/apache/beam/issues/24971#issuecomment-1426662159
Correct me if I'm wrong, but in a bundle, when a message is failed the whole bundle will be retried which is not what we want too also, the exception was catched in processElement which makes the DoFn to keep the same instance of the publisher with a failed connection.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org