You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@beam.apache.org by Wiśniowski Piotr <co...@gmail.com> on 2023/10/02 20:47:28 UTC
Re: UDF/UADF over complex structures
Hi Gyorgy,
I guess Your problem might not be directly related to UDFs not UDAFs but
to nested structure of Your data. I did have a problem with processing
nested data and also did not find a way to work this out - especially if
You are not in control of input data structure.
This issues seem very related as nested Rows need some polishing:
- https://github.com/apache/beam/issues/26911
- https://github.com/apache/beam/issues/27733
I have plans to jump on fix for this two, and I guess it would not be
hard, but I do not have any capacity for open source tasks for now.
Anyone would like to take a look at them?
Best
Wiśniowski Piotr
On 28.09.2023 19:31, Balogh, György wrote:
> Sorry I was not specific enough. I ment using the SqlTransform
> registerUdf and registerUdaf. I use a lot of SQL in my pipeline and I
> would prefer using SQL UDFs in many cases over writing beam
> transforms. I already have UDFs but I did not find a way to make them
> work over nested structures.
> Thank you,
> Gyorgy
>
> On Thu, Sep 28, 2023 at 5:40 PM Robert Bradshaw via user
> <us...@beam.apache.org> wrote:
>
> Yes, for sure. This is one of the areas Beam excels vs.
> more simple tools like SQL. You can write arbitrary code to
> iterate over arbitrary structures in the typical
> Java/Python/Go/Typescript/Scala/[pick your language] way. In the
> Beam nomenclature. UDFs correspond to DoFns and UDAFs correspond
> to CombineFns.
>
> On Thu, Sep 28, 2023 at 4:23 AM Balogh, György
> <bo...@ultinous.com> wrote:
>
> Hi,
> I've complex nested structure in my input data. Is it possible
> to have UDF/UDAF taking nested structure as input? I'm using
> java. Outputting nested structure is also a question.
> Thank you,
> Gyorgy
> --
>
> György Balogh
> CTO
> E gyorgy.balogh@ultinous.com <ma...@ultinous.com>
> M +36 30 270 8342 <tel:+36%2030%20270%208342>
> A HU, 1117 Budapest, Budafoki út 209.
> W www.ultinous.com <http://www.ultinous.com>
>
>
>
> --
>
> György Balogh
> CTO
> E gyorgy.balogh@ultinous.com <ma...@ultinous.com>
> M +36 30 270 8342 <tel:+36%2030%20270%208342>
> A HU, 1117 Budapest, Budafoki út 209.
> W www.ultinous.com <http://www.ultinous.com>
>