Any caveats about processing abstract classes ?

[Davide Bolcioni via user <us...@flink.apache.org>]

Greetings,
I am looking at Flink pipeline processing events consumed from a Kafka
topic, which now needs to also consume events which have a different, but
related, schema. Traditional Java OOP would suggest transitioning from

class Dog { ... }
new FilterFunction<Dog> { ... }

to

abstract class Animal { ... }
class Dog extends Animal { ... }
class Cat extends Animal { ... }
new FilterFunction<Animal> { ... }

but I am wondering if there is anything that might surprise the unwary down
that road, considering that the code base also uses asynchronous functions
and the broadcast pattern.

Thank you in advance,
Davide Bolcioni
--
There is no place like /home

Re: Any caveats about processing abstract classes ?

[Gen Luo <lu...@gmail.com>]

Sorry but I'm not really familiar about the interface. As far as I can see,
it's mainly used in the DataSet API. Most of its implementations provide a
TypeSerializer, like `DoubleValueSerializer`. That works the same way as a
KryoSerializer or a customized serializer. So I suppose you don't have to
use it.

On Sat, Nov 12, 2022 at 8:24 AM Davide Bolcioni via user <
user@flink.apache.org> wrote:

> Thanks,
> On Thu, Nov 10, 2022 at 6:21 PM Gen Luo <lu...@gmail.com> wrote
>
>>
>> I suppose it would be fine. The only difference I can figure out that may
>> matter is the serialization. Flink uses KryoSerializer as the fallback
>> serializer if the TypeInformation of the records is not provided, which can
>> properly process abstract classes. This works well in most cases.
>>
>> For better compatibility you may want to use a customized serializer. In
>> such cases you can call SingleOutputStreamOperator#returns(TypeInformation)
>> with your TypeInformation like:
>>     input.map(new MapToAnimal()).returns(new AnimalTypeInformation())
>>
>
> I noticed that implementing org.apache.flink.types.Value interface is
> listed as an option for serialization, is that a superior alternative? Or
> is it for the table API only ? Are there pros and cons ?
>
> Davide Bolcioni--
> There is no place like /home
>

Re: Any caveats about processing abstract classes ?

[Davide Bolcioni via user <us...@flink.apache.org>]

Thanks,
On Thu, Nov 10, 2022 at 6:21 PM Gen Luo <lu...@gmail.com> wrote

>
> I suppose it would be fine. The only difference I can figure out that may
> matter is the serialization. Flink uses KryoSerializer as the fallback
> serializer if the TypeInformation of the records is not provided, which can
> properly process abstract classes. This works well in most cases.
>
> For better compatibility you may want to use a customized serializer. In
> such cases you can call SingleOutputStreamOperator#returns(TypeInformation)
> with your TypeInformation like:
>     input.map(new MapToAnimal()).returns(new AnimalTypeInformation())
>

I noticed that implementing org.apache.flink.types.Value interface is
listed as an option for serialization, is that a superior alternative? Or
is it for the table API only ? Are there pros and cons ?

Davide Bolcioni--
There is no place like /home

Re: Any caveats about processing abstract classes ?

[Gen Luo <lu...@gmail.com>]

Hi Davide,

I suppose it would be fine. The only difference I can figure out that may
matter is the serialization. Flink uses KryoSerializer as the fallback
serializer if the TypeInformation of the records is not provided, which can
properly process abstract classes. This works well in most cases.

For better compatibility you may want to use a customized serializer. In
such cases you can call SingleOutputStreamOperator#returns(TypeInformation)
with your TypeInformation like:
    input.map(new MapToAnimal()).returns(new AnimalTypeInformation())


On Thu, Nov 10, 2022 at 9:02 AM Davide Bolcioni via user <
user@flink.apache.org> wrote:

> Greetings,
> I am looking at Flink pipeline processing events consumed from a Kafka
> topic, which now needs to also consume events which have a different, but
> related, schema. Traditional Java OOP would suggest transitioning from
>
> class Dog { ... }
> new FilterFunction<Dog> { ... }
>
> to
>
> abstract class Animal { ... }
> class Dog extends Animal { ... }
> class Cat extends Animal { ... }
> new FilterFunction<Animal> { ... }
>
> but I am wondering if there is anything that might surprise the unwary
> down that road, considering that the code base also uses asynchronous
> functions and the broadcast pattern.
>
> Thank you in advance,
> Davide Bolcioni
> --
> There is no place like /home
>