You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@beam.apache.org by Shivam Singhal <sh...@gmail.com> on 2023/04/05 11:31:04 UTC

How to handle errors in GO SDK in a custom PTransform

Hi folks,

In Java SDK, we have robust error handling using tagged outputs and
PCollectionTuples.

Do we have something similar in Go SDK? I have been unable to locate it in
the reference docs
<https://pkg.go.dev/github.com/apache/beam/sdks/v2/go/pkg/beam>.

*A general usecase for error handling who might not be familiar with error
handling in Java SDK:*
My custom PTransform can throw error while writing to Redis and I need to
know which key-value pairs were not written in the redis. My PTransform
will also output the successfully written key-value pairs. I need a way to
somehow differentiate between successful and error outputs.

Re: How to handle errors in GO SDK in a custom PTransform

Posted by Shivam Singhal <sh...@gmail.com>.

Thanks, Danny! This was what I was looking for.

On Wed, 5 Apr 2023 at 18:40, Danny McCormick via user <us...@beam.apache.org>
wrote:

> The Go SDK doesn't use tagged outputs, instead it uses positional ordering
> for emitting multiple outputs. So you can do something like:
>
> func processElements(element string, goodEmit, errEmit func(string)) {
>    if element.isGood {
>       goodEmit(element)
>    } else {
>       errEmit(element)
>    }
> }
>
> which could then be consumed as:
>
> goodElements, badElements := beam.ParDo2(s, processElements, inputElements)
>
> See https://beam.apache.org/documentation/programming-guide/#output-tags
> for more details.
>
> Thanks,
> Danny
>
> On Wed, Apr 5, 2023 at 7:31 AM Shivam Singhal <sh...@gmail.com>
> wrote:
>
>> Hi folks,
>>
>> In Java SDK, we have robust error handling using tagged outputs and
>> PCollectionTuples.
>>
>> Do we have something similar in Go SDK? I have been unable to locate it
>> in the reference docs
>> <https://pkg.go.dev/github.com/apache/beam/sdks/v2/go/pkg/beam>.
>>
>> *A general usecase for error handling who might not be familiar with
>> error handling in Java SDK:*
>> My custom PTransform can throw error while writing to Redis and I need to
>> know which key-value pairs were not written in the redis. My PTransform
>> will also output the successfully written key-value pairs. I need a way to
>> somehow differentiate between successful and error outputs.
>>
>>

Re: How to handle errors in GO SDK in a custom PTransform

Posted by Danny McCormick via user <us...@beam.apache.org>.

The Go SDK doesn't use tagged outputs, instead it uses positional ordering
for emitting multiple outputs. So you can do something like:

func processElements(element string, goodEmit, errEmit func(string)) {
   if element.isGood {
      goodEmit(element)
   } else {
      errEmit(element)
   }
}

which could then be consumed as:

goodElements, badElements := beam.ParDo2(s, processElements, inputElements)

See https://beam.apache.org/documentation/programming-guide/#output-tags
for more details.

Thanks,
Danny

On Wed, Apr 5, 2023 at 7:31 AM Shivam Singhal <sh...@gmail.com>
wrote:

> Hi folks,
>
> In Java SDK, we have robust error handling using tagged outputs and
> PCollectionTuples.
>
> Do we have something similar in Go SDK? I have been unable to locate it in
> the reference docs
> <https://pkg.go.dev/github.com/apache/beam/sdks/v2/go/pkg/beam>.
>
> *A general usecase for error handling who might not be familiar with error
> handling in Java SDK:*
> My custom PTransform can throw error while writing to Redis and I need to
> know which key-value pairs were not written in the redis. My PTransform
> will also output the successfully written key-value pairs. I need a way to
> somehow differentiate between successful and error outputs.
>
>