You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "xuanswe (via GitHub)" <gi...@apache.org> on 2025/02/15 21:25:00 UTC
[I] [Feature Request]: PubSubIO should support disabling auto ack after the 1st fused stage [beam]
xuanswe opened a new issue, #34000:
URL: https://github.com/apache/beam/issues/34000
### What would you like to happen?
According to https://cloud.google.com/dataflow/docs/concepts/streaming-with-cloud-pubsub#dead-letter-topics
> Dataflow might acknowledge messages before the pipeline fully processes the data. Specifically, Dataflow acknowledges messages after they are successfully processed by the first fused stage and side effects of that processing have been written to persistent storage. If the pipeline has multiple fused stages and failures occur at any point after the first stage, the messages are already acknowledged and don't go to the dead-letter topic.
First of all, I am new to Dataflow and PubSubIO, so if I am wrong on something, please correct me.
I understand that, if there is an uncaught exception (ex. due to a bug) after the first fused stage, the message is already acked and will never be redelivered after fixing the bug.
If I am wrong, please explain to me where I am wrong and please explain the way dataflow prevents data loss in this situation.
If it's true, could you please tell me the solution to ensure **100%** that I do not lose any data for the mentioned situation or anything?
If there is currently no solution yet. What do you think if PubSubIO supports a method `withAutoAckOnSuccess(false)` like below?
```
PubsubIO.readMessagesWithAttributes()
.fromSubscription("projects/projectId/subscriptions/subscriptionId")
.withAutoAckOnSuccess(false)
```
Then developer will manually ack the message at the last stage of the pipeline manually.
### Issue Priority
Priority: 2 (default / most feature requests should be filed as P2)
### Issue Components
- [ ] Component: Python SDK
- [ ] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [x] Component: IO connector
- [ ] Component: Beam YAML
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Infrastructure
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Samza Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [x] Component: Google Cloud Dataflow Runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Request]: PubSubIO should allow disabling auto ack after the 1st fused stage [beam]
Posted by "xuanswe (via GitHub)" <gi...@apache.org>.
xuanswe commented on issue #34000:
URL: https://github.com/apache/beam/issues/34000#issuecomment-2661545053
I don't understand what you really mean. Could you please explain with more details?
I drained my old job with uncaught exception. This job doesn't have dead-letter pattern.
Now, I don't know how to start a new job and get back the unprocessed message in the new job.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Request]: PubSubIO should allow disabling auto ack after the 1st fused stage [beam]
Posted by "liferoad (via GitHub)" <gi...@apache.org>.
liferoad commented on issue #34000:
URL: https://github.com/apache/beam/issues/34000#issuecomment-2661551417
https://cloud.google.com/dataflow/docs/guides/stopping-a-pipeline#drain
If any exception happens during the draining, the job would be stuck. Without knowing your Dataflow job details, it will be hard to know what went wrong. You probably should open a Dataflow support ticket.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Request]: PubSubIO should allow disabling auto ack after the 1st fused stage [beam]
Posted by "liferoad (via GitHub)" <gi...@apache.org>.
liferoad commented on issue #34000:
URL: https://github.com/apache/beam/issues/34000#issuecomment-2661475641
For your old job, you should be able to drain it without any data loss.https://cloud.google.com/dataflow/docs/concepts/streaming-with-cloud-pubsub#dead-letter-topics also mentions that do not use the pub/sub dead letter topics with Dataflow. You should implement the dead-letter pattern explicitly in your pipeline.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Request]: PubSubIO should allow disabling auto ack after the 1st fused stage [beam]
Posted by "xuanswe (via GitHub)" <gi...@apache.org>.
xuanswe commented on issue #34000:
URL: https://github.com/apache/beam/issues/34000#issuecomment-2661376760
.add-labels pubsub
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Request]: PubSubIO should allow disabling auto ack after the 1st fused stage [beam]
Posted by "xuanswe (via GitHub)" <gi...@apache.org>.
xuanswe closed issue #34000: [Feature Request]: PubSubIO should allow disabling auto ack after the 1st fused stage
URL: https://github.com/apache/beam/issues/34000
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Request]: PubSubIO should allow disabling auto ack after the 1st fused stage [beam]
Posted by "xuanswe (via GitHub)" <gi...@apache.org>.
xuanswe commented on issue #34000:
URL: https://github.com/apache/beam/issues/34000#issuecomment-2661572572
> If any exception happens during the draining, the job would be stuck.
Yes, it was stuck. I am simulating the situation again and the job cannot even drain :).
> You probably should open a Dataflow support ticket.
I am only simulating the situation with my personal free account.
> Without knowing your Dataflow job details, it will be hard to know what went wrong.
The scenario is simple: read a message from PubSub using PubSubIO then throw an exception after the first fused stage.
The point is that, user needs to drain the job and do a lot of stuff in order to prevent the loss of PubSub messages.
I open this ticket is to show that for some common situations, there is a better way.
If the pipeline is idempotent, we don't need to care about draining at all.
We can just disable the auto ack of PubSubIO and manually ack at the end of the pipeline, then the problem is solved.
No more draining, no more worrying about how dataflow works.
We just let it crash and deploy a new job. Don't care about the old job.
Because the unprocessed messages are not yet acked, so we will get them again in the new job.
So, my proposal in this ticket is to support something like `.withAutoAckOnSuccess(false)` in PubsubIO.
What is your opinion?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Request]: PubSubIO should allow disabling auto ack after the 1st fused stage [beam]
Posted by "xuanswe (via GitHub)" <gi...@apache.org>.
xuanswe commented on issue #34000:
URL: https://github.com/apache/beam/issues/34000#issuecomment-2661473959
Hi @liferoad,
> You won't be able to update the job if your new code is not compatible with the existing pipeline
I know that, and that's the issue.
I would like to cancel the old job and deploy the new job. But the message is acked and unprocessed in the old job.
It will not be redelivered to the new job, hence losing data.
If I am able to control ack manually, I will not lose data and freely deploy a new job and don't care what happens in the old job.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Request]: PubSubIO should allow disabling auto ack after the 1st fused stage [beam]
Posted by "liferoad (via GitHub)" <gi...@apache.org>.
liferoad commented on issue #34000:
URL: https://github.com/apache/beam/issues/34000#issuecomment-2661471125
You won't be able to update the job if your new code is not compatible with the existing pipeline: https://cloud.google.com/dataflow/docs/guides/updating-a-pipeline. Dataflow performs a compatibility check between your currently running job and your potential replacement job.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Request]: PubSubIO should allow disabling auto ack after the 1st fused stage [beam]
Posted by "liferoad (via GitHub)" <gi...@apache.org>.
liferoad commented on issue #34000:
URL: https://github.com/apache/beam/issues/34000#issuecomment-2661712249
https://cloud.google.com/dataflow/docs/guides/templates/provided/pubsub-subscription-to-bigquery#template-source-code shows how to handle any failed rows. The source code is here: https://github.com/GoogleCloudPlatform/DataflowTemplates/blob/main/v1/src/main/java/com/google/cloud/teleport/templates/PubSubToBigQuery.java#L281.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Request]: PubSubIO should allow disabling auto ack after the 1st fused stage [beam]
Posted by "wattache (via GitHub)" <gi...@apache.org>.
wattache commented on issue #34000:
URL: https://github.com/apache/beam/issues/34000#issuecomment-3001020034
Hello @xuanswe
From what I understood your question was about PubSub and not just handling failed inserts in BigQuery ? I have the same issue when dealing with a custom ClickHouse connector.
As far as I understand the method [finalizeCheckpoint](https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubUnboundedSource.java#L267) is called. It seems to come from [Read.java](https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/io/Read.java#L612) when setting a bundle finalizer.
So two observations:
1. it seems hard to change this behaviour with a quick win
2. the fact `bundleFinalizer` is called indeed suggests a fused step is made, when the fused step finished processing the bundle it's acked, and then any message between here and final destination (ClickHouse for me) is eventually lost
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org