You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@beam.apache.org by John Casey via dev <de...@beam.apache.org> on 2023/11/08 20:43:39 UTC

Adding Dead Letter Queues to Beam IOs

Hi All,

I've written up a design for adding DLQs to existing Beam IOs. It's been
through a round of reviews with some Dataflow folks at Google, but I'd
appreciate any comments the rest of Beam have around how to refine the
design.

TL;DR: Make it easy for a user to configure IOs to route bad data to an
alternate sink instead of crashing the pipeline or having the record be
retried indefinitely.

https://docs.google.com/document/d/1NGeCk6tOqF-TiGEAV7ixd_vhIiWz9sHPlCa1P_77Ajs/edit?usp=sharing

Thanks!

John

Re: Adding Dead Letter Queues to Beam IOs

Posted by Austin Bennett <au...@apache.org>.

This will eventually be a great addition to aid usability -- thanks for
starting to think through and address, John!



On Fri, Nov 10, 2023, 10:54 AM Danny McCormick via dev <de...@beam.apache.org>
wrote:

> Thanks - the general ideas seem solid, I added some questions/comments as
> well.
>
> On Fri, Nov 10, 2023 at 1:32 PM Robert Bradshaw via dev <
> dev@beam.apache.org> wrote:
>
>> Thanks. I added some comments to the doc and open PR.
>>
>> On Wed, Nov 8, 2023 at 12:44 PM John Casey via dev <de...@beam.apache.org>
>> wrote:
>> >
>> > Hi All,
>> >
>> > I've written up a design for adding DLQs to existing Beam IOs. It's
>> been through a round of reviews with some Dataflow folks at Google, but I'd
>> appreciate any comments the rest of Beam have around how to refine the
>> design.
>> >
>> > TL;DR: Make it easy for a user to configure IOs to route bad data to an
>> alternate sink instead of crashing the pipeline or having the record be
>> retried indefinitely.
>> >
>> >
>> https://docs.google.com/document/d/1NGeCk6tOqF-TiGEAV7ixd_vhIiWz9sHPlCa1P_77Ajs/edit?usp=sharing
>> >
>> > Thanks!
>> >
>> > John
>>
>

Re: Adding Dead Letter Queues to Beam IOs

Posted by Danny McCormick via dev <de...@beam.apache.org>.

Thanks - the general ideas seem solid, I added some questions/comments as
well.

On Fri, Nov 10, 2023 at 1:32 PM Robert Bradshaw via dev <de...@beam.apache.org>
wrote:

> Thanks. I added some comments to the doc and open PR.
>
> On Wed, Nov 8, 2023 at 12:44 PM John Casey via dev <de...@beam.apache.org>
> wrote:
> >
> > Hi All,
> >
> > I've written up a design for adding DLQs to existing Beam IOs. It's been
> through a round of reviews with some Dataflow folks at Google, but I'd
> appreciate any comments the rest of Beam have around how to refine the
> design.
> >
> > TL;DR: Make it easy for a user to configure IOs to route bad data to an
> alternate sink instead of crashing the pipeline or having the record be
> retried indefinitely.
> >
> >
> https://docs.google.com/document/d/1NGeCk6tOqF-TiGEAV7ixd_vhIiWz9sHPlCa1P_77Ajs/edit?usp=sharing
> >
> > Thanks!
> >
> > John
>

Re: Adding Dead Letter Queues to Beam IOs

Posted by Robert Bradshaw via dev <de...@beam.apache.org>.

Thanks. I added some comments to the doc and open PR.

On Wed, Nov 8, 2023 at 12:44 PM John Casey via dev <de...@beam.apache.org> wrote:
>
> Hi All,
>
> I've written up a design for adding DLQs to existing Beam IOs. It's been through a round of reviews with some Dataflow folks at Google, but I'd appreciate any comments the rest of Beam have around how to refine the design.
>
> TL;DR: Make it easy for a user to configure IOs to route bad data to an alternate sink instead of crashing the pipeline or having the record be retried indefinitely.
>
> https://docs.google.com/document/d/1NGeCk6tOqF-TiGEAV7ixd_vhIiWz9sHPlCa1P_77Ajs/edit?usp=sharing
>
> Thanks!
>
> John

Re: Adding Dead Letter Queues to Beam IOs

Posted by Alexey Romanenko <ar...@gmail.com>.

Thanks a lot for working on this, long waiting and very demanded user feature.

I’ll try to take a look on design doc in the next days.

—
Alexey

> On 8 Nov 2023, at 21:43, John Casey via dev <de...@beam.apache.org> wrote:
> 
> Hi All,
> 
> I've written up a design for adding DLQs to existing Beam IOs. It's been through a round of reviews with some Dataflow folks at Google, but I'd appreciate any comments the rest of Beam have around how to refine the design.
> 
> TL;DR: Make it easy for a user to configure IOs to route bad data to an alternate sink instead of crashing the pipeline or having the record be retried indefinitely.
> 
> https://docs.google.com/document/d/1NGeCk6tOqF-TiGEAV7ixd_vhIiWz9sHPlCa1P_77Ajs/edit?usp=sharing
> 
> Thanks!
> 
> John