You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@beam.apache.org by Rener Pereira De Castro via user <us...@beam.apache.org> on 2022/09/16 11:43:32 UTC

[Question] [Go SDK] Generic Register DoFn having iter(*interface{})

Hi guys,


I can't find a way to generic register a DoFn with ProcessElement like
parquetWriteFn -  ProcessElement(ctx context.Context, _ int, iter func(*interface{}) bool) (https://github.com/apache/beam/blob/30a48f05cf2ee0eea0a304fea01eb40f323f9f3c/sdks/go/pkg/beam/io/parquetio/parquetio.go#L150)

I'm having this error:
```
bad input type
        caused by:
        encoding full type interface {}
bad type
        caused by:
unencodable type interface {}
```

Any help on that will be greatly appreciated


Regards,

[cid:e17aa8cb-298b-41ba-b189-03e528e9c676]

Rener Castro
Data & Analytics
email: rener.castro@c6bank.com<ma...@c6bank.com>

[signature_1033755691]



Classificação: Interna


Classificação: Interna

Re: [Question] [Go SDK] Generic Register DoFn having iter(*interface{})

Posted by Rener Pereira De Castro via user <us...@beam.apache.org>.
Hi Danny,

Thank you for answer.

This code below don’t work for me:

register.Iter1[interface{}]()
register.DoFn3x1[context.Context, int, func(*interface{}) bool, error](&writeParquetFn{})

I'm having this error when trying to run on Dataflow:
```
bad input type
        caused by:
        encoding full type interface {}
bad type
        caused by:
unencodable type interface {}
```

Regards,

[Gif com o logo do C6 Bank em fundo preto]<https://www.c6bank.com.br/>
Rener Castro
Dados & Analytics
phone: +­55 11 9 8700.0199
www.c6bank.com.br<https://www.c6bank.com.br/>


From: Danny McCormick <da...@google.com>
Date: Friday, 16 September 2022 10:08
To: user@beam.apache.org <us...@beam.apache.org>, Rener Pereira De Castro <Re...@c6bank.com>
Subject: Re: [Question] [Go SDK] Generic Register DoFn having iter(*interface{})
Hey Rener, you should be able to register that function with `register.DoFn3x1[context.Context, int, func(*interface{}) bool, error]`. You would use DoFn3x1 because you have 3 inputs and 1 output in your ProcessElement, and then the constraints are just the input types to your ProcessElement function.

Thanks,
Danny

On Fri, Sep 16, 2022 at 7:44 AM Rener Pereira De Castro via user <us...@beam.apache.org>> wrote:
Hi guys,


I can't find a way to generic register a DoFn with ProcessElement like
parquetWriteFn -  ProcessElement(ctx context.Context, _ int, iter func(*interface{}) bool) (https://github.com/apache/beam/blob/30a48f05cf2ee0eea0a304fea01eb40f323f9f3c/sdks/go/pkg/beam/io/parquetio/parquetio.go#L150<https://protect2.fireeye.com/v1/url?k=31323334-501cfaeb-31350c47-454455535732-783c0a34be56f4a0&q=1&e=163b1df8-451a-4632-b4f9-df898c6a9902&u=https%3A%2F%2Fgithub.com%2Fapache%2Fbeam%2Fblob%2F30a48f05cf2ee0eea0a304fea01eb40f323f9f3c%2Fsdks%2Fgo%2Fpkg%2Fbeam%2Fio%2Fparquetio%2Fparquetio.go%23L150>)

I'm having this error:
```
bad input type
        caused by:
        encoding full type interface {}
bad type
        caused by:
unencodable type interface {}
```

Any help on that will be greatly appreciated


Regards,

[cid:1834667b7a75f2c7571]

Rener Castro
Data & Analytics
email: rener.castro@c6bank.com<ma...@c6bank.com>

[signature_1033755691]



Classificação: Interna


Classificação: Interna


Classificação: Interna


Classificação: Interna

Re: [Question] [Go SDK] Generic Register DoFn having iter(*interface{})

Posted by Danny McCormick via user <us...@beam.apache.org>.
Ah, I see - I didn't realize the problem was happening post-registration.
This error originates from
https://github.com/apache/beam/blob/8b2676782a62f8bdf912395267056c9f37251338/sdks/go/pkg/beam/core/runtime/graphx/serialize.go#L502
and basically means that we are not able to infer a default coder for your
type (in this case *interface{}). More information about coders can be
found at
https://beam.apache.org/documentation/programming-guide/#data-encoding-and-type-safety

Normally, the correct way to resolve this would be to register a custom
coder
<https://beam.apache.org/documentation/programming-guide/#setting-default-coder>
for
*interface{} -
https://beam.apache.org/documentation/programming-guide/#setting-default-coder
- could you try doing that?

On Fri, Sep 16, 2022 at 9:28 AM Rener Pereira De Castro <
Rener.Castro@c6bank.com> wrote:

> Hi Danny,
>
>
>
> Thank you for your answer.
>
>
>
> This code below doesn’t work for me:
>
>
>
> *register.Iter1[interface{}]() *
>
> *register.DoFn3x1[context.Context, int, func(*interface{}) bool,
> error](&writeParquetFn{}) *
>
>
>
> I'm having this error when trying to run on Dataflow:
>
> ```
>
> bad input type
>
>         caused by:
>
>         encoding full type interface {}
>
> bad type
>
>         caused by:
>
> unencodable type interface {}
>
> ```
>
>
>
> Regards,
>
> ------------------------------
> *From:* Danny McCormick <da...@google.com>
> *Sent:* Friday, September 16, 2022 10:07 AM
> *To:* user@beam.apache.org <us...@beam.apache.org>; Rener Pereira De
> Castro <Re...@c6bank.com>
> *Subject:* Re: [Question] [Go SDK] Generic Register DoFn having
> iter(*interface{})
>
> Hey Rener, you should be able to register that function with
> `register.DoFn3x1[context.Context, int, func(*interface{}) bool, error]`.
> You would use DoFn3x1 because you have 3 inputs and 1 output in your
> ProcessElement, and then the constraints are just the input types to your
> ProcessElement function.
>
> Thanks,
> Danny
>
> On Fri, Sep 16, 2022 at 7:44 AM Rener Pereira De Castro via user <
> user@beam.apache.org> wrote:
>
> Hi guys,
>
>
> I can't find a way to generic register a DoFn with ProcessElement like
> *parquetWriteFn -  ProcessElement(ctx context.Context, _ int, iter
> func(*interface{}) bool)
> (https://github.com/apache/beam/blob/30a48f05cf2ee0eea0a304fea01eb40f323f9f3c/sdks/go/pkg/beam/io/parquetio/parquetio.go#L150
> <https://protect2.fireeye.com/v1/url?k=31323334-501cfaeb-31350c47-454455535732-783c0a34be56f4a0&q=1&e=163b1df8-451a-4632-b4f9-df898c6a9902&u=https%3A%2F%2Fgithub.com%2Fapache%2Fbeam%2Fblob%2F30a48f05cf2ee0eea0a304fea01eb40f323f9f3c%2Fsdks%2Fgo%2Fpkg%2Fbeam%2Fio%2Fparquetio%2Fparquetio.go%23L150>)*
>
> I'm having this error:
> ```
> bad input type
>         caused by:
>         encoding full type interface {}
> bad type
>         caused by:
> unencodable type interface {}
> ```
>
> Any help on that will be greatly appreciated
>
>
> Regards,
>
> *Rener Castro*
> *Data & Analytics*
> email: *rener.castro@c6bank.com* <br...@c6bank.com>
>
> [image: signature_1033755691]
>
>
> Classificação: Interna
>
> Classificação: Interna
>
>
> Classificação: Interna
>
> Classificação: Interna
>

Re: [Question] [Go SDK] Generic Register DoFn having iter(*interface{})

Posted by Rener Pereira De Castro via user <us...@beam.apache.org>.
Hi Danny,

Thank you for your answer.

This code below doesn’t work for me:

register.Iter1[interface{}]()
register.DoFn3x1[context.Context, int, func(*interface{}) bool, error](&writeParquetFn{})

I'm having this error when trying to run on Dataflow:
```
bad input type
        caused by:
        encoding full type interface {}
bad type
        caused by:
unencodable type interface {}
```

Regards,

________________________________
From: Danny McCormick <da...@google.com>
Sent: Friday, September 16, 2022 10:07 AM
To: user@beam.apache.org <us...@beam.apache.org>; Rener Pereira De Castro <Re...@c6bank.com>
Subject: Re: [Question] [Go SDK] Generic Register DoFn having iter(*interface{})

Hey Rener, you should be able to register that function with `register.DoFn3x1[context.Context, int, func(*interface{}) bool, error]`. You would use DoFn3x1 because you have 3 inputs and 1 output in your ProcessElement, and then the constraints are just the input types to your ProcessElement function.

Thanks,
Danny

On Fri, Sep 16, 2022 at 7:44 AM Rener Pereira De Castro via user <us...@beam.apache.org>> wrote:
Hi guys,


I can't find a way to generic register a DoFn with ProcessElement like
parquetWriteFn -  ProcessElement(ctx context.Context, _ int, iter func(*interface{}) bool) (https://github.com/apache/beam/blob/30a48f05cf2ee0eea0a304fea01eb40f323f9f3c/sdks/go/pkg/beam/io/parquetio/parquetio.go#L150<https://protect2.fireeye.com/v1/url?k=31323334-501cfaeb-31350c47-454455535732-783c0a34be56f4a0&q=1&e=163b1df8-451a-4632-b4f9-df898c6a9902&u=https%3A%2F%2Fgithub.com%2Fapache%2Fbeam%2Fblob%2F30a48f05cf2ee0eea0a304fea01eb40f323f9f3c%2Fsdks%2Fgo%2Fpkg%2Fbeam%2Fio%2Fparquetio%2Fparquetio.go%23L150>)

I'm having this error:
```
bad input type
        caused by:
        encoding full type interface {}
bad type
        caused by:
unencodable type interface {}
```

Any help on that will be greatly appreciated


Regards,

[cid:1834667b7a75f2c7571]

Rener Castro
Data & Analytics
email: rener.castro@c6bank.com<ma...@c6bank.com>

[signature_1033755691]



Classificação: Interna


Classificação: Interna


Classificação: Interna


Classificação: Interna

Re: [Question] [Go SDK] Generic Register DoFn having iter(*interface{})

Posted by Rener Pereira De Castro via user <us...@beam.apache.org>.
Hi Danny,

Thank you for answer.

This code below don’t work for me:

register.Iter1[interface{}]()
register.DoFn3x1[context.Context, int, func(*interface{}) bool, error](&writeParquetFn{})

I'm having this error when trying to run on Dataflow:
```
bad input type
        caused by:
        encoding full type interface {}
bad type
        caused by:
unencodable type interface {}
```

Regards,


[Gif com o logo do C6 Bank em fundo preto]<https://www.c6bank.com.br/>
Rener Castro
Dados & Analytics
phone: +­55 11 9 8700.0199
www.c6bank.com.br<https://www.c6bank.com.br/>


From: Danny McCormick <da...@google.com>
Date: Friday, 16 September 2022 10:08
To: user@beam.apache.org <us...@beam.apache.org>, Rener Pereira De Castro <Re...@c6bank.com>
Subject: Re: [Question] [Go SDK] Generic Register DoFn having iter(*interface{})
Hey Rener, you should be able to register that function with `register.DoFn3x1[context.Context, int, func(*interface{}) bool, error]`. You would use DoFn3x1 because you have 3 inputs and 1 output in your ProcessElement, and then the constraints are just the input types to your ProcessElement function.

Thanks,
Danny

On Fri, Sep 16, 2022 at 7:44 AM Rener Pereira De Castro via user <us...@beam.apache.org>> wrote:
Hi guys,


I can't find a way to generic register a DoFn with ProcessElement like
parquetWriteFn -  ProcessElement(ctx context.Context, _ int, iter func(*interface{}) bool) (https://github.com/apache/beam/blob/30a48f05cf2ee0eea0a304fea01eb40f323f9f3c/sdks/go/pkg/beam/io/parquetio/parquetio.go#L150<https://protect2.fireeye.com/v1/url?k=31323334-501cfaeb-31350c47-454455535732-783c0a34be56f4a0&q=1&e=163b1df8-451a-4632-b4f9-df898c6a9902&u=https%3A%2F%2Fgithub.com%2Fapache%2Fbeam%2Fblob%2F30a48f05cf2ee0eea0a304fea01eb40f323f9f3c%2Fsdks%2Fgo%2Fpkg%2Fbeam%2Fio%2Fparquetio%2Fparquetio.go%23L150>)

I'm having this error:
```
bad input type
        caused by:
        encoding full type interface {}
bad type
        caused by:
unencodable type interface {}
```

Any help on that will be greatly appreciated


Regards,

[cid:1834667b7a75f2c7571]

Rener Castro
Data & Analytics
email: rener.castro@c6bank.com<ma...@c6bank.com>

[signature_1033755691]



Classificação: Interna


Classificação: Interna


Classificação: Interna


Classificação: Interna

Re: [Question] [Go SDK] Generic Register DoFn having iter(*interface{})

Posted by Danny McCormick via user <us...@beam.apache.org>.
Hey Rener, you should be able to register that function with
`register.DoFn3x1[context.Context, int, func(*interface{}) bool, error]`.
You would use DoFn3x1 because you have 3 inputs and 1 output in your
ProcessElement, and then the constraints are just the input types to your
ProcessElement function.

Thanks,
Danny

On Fri, Sep 16, 2022 at 7:44 AM Rener Pereira De Castro via user <
user@beam.apache.org> wrote:

> Hi guys,
>
>
> I can't find a way to generic register a DoFn with ProcessElement like
> *parquetWriteFn -  ProcessElement(ctx context.Context, _ int, iter
> func(*interface{}) bool)
> (https://github.com/apache/beam/blob/30a48f05cf2ee0eea0a304fea01eb40f323f9f3c/sdks/go/pkg/beam/io/parquetio/parquetio.go#L150
> <https://github.com/apache/beam/blob/30a48f05cf2ee0eea0a304fea01eb40f323f9f3c/sdks/go/pkg/beam/io/parquetio/parquetio.go#L150>)*
>
> I'm having this error:
> ```
> bad input type
>         caused by:
>         encoding full type interface {}
> bad type
>         caused by:
> unencodable type interface {}
> ```
>
> Any help on that will be greatly appreciated
>
>
> Regards,
>
> *Rener Castro*
> *Data & Analytics*
> email: *rener.castro@c6bank.com* <br...@c6bank.com>
>
> [image: signature_1033755691]
>
>
> Classificação: Interna
>
> Classificação: Interna
>