You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@beam.apache.org by Utkarsh Parekh <ut...@gmail.com> on 2024/12/20 17:02:16 UTC

Commit Kafka message after processing

Hello everyone,

For Python SDK users, what configuration do you use in the beam pipeline to
ensure committing the offset after loading data in BigQuery?

What's the best approach?

Utkarsh

Re: Commit Kafka message after processing

Posted by XQ Hu via user <us...@beam.apache.org>.

Assuming you are using Dataflow as your runner, please check
https://cloud.google.com/dataflow/docs/guides/read-from-kafka

On Fri, Dec 20, 2024 at 12:03 PM Utkarsh Parekh <ut...@gmail.com>
wrote:

> Hello everyone,
>
> For Python SDK users, what configuration do you use in the beam pipeline
> to ensure committing the offset after loading data in BigQuery?
>
> What's the best approach?
>
> Utkarsh
>
>
>