You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by David Radley <da...@uk.ibm.com> on 2024/05/01 13:47:05 UTC

FW: RE: [DISCUSS] FLIP-XXX Apicurio-avro format

Hi Martijn,
Any news?
           Kind regards, David.


From: David Radley <da...@uk.ibm.com>
Date: Monday, 22 April 2024 at 09:48
To: dev@flink.apache.org <de...@flink.apache.org>
Subject: FW: [EXTERNAL] RE: [DISCUSS] FLIP-XXX Apicurio-avro format
Hi Martijn,
A gentle nudge, is this ok for you or one of the PMC or committers to create a Flip now?
           Kind regards, David.

From: David Radley <da...@uk.ibm.com>
Date: Monday, 15 April 2024 at 12:29
To: dev@flink.apache.org <de...@flink.apache.org>
Subject: Re: [EXTERNAL] RE: [DISCUSS] FLIP-XXX Apicurio-avro format
Hi Martijn,
Thanks for looking at this. I have used the template in a new  Google Doc https://docs.google.com/document/d/1J1E-cE-X2H3-kw4rNjLn71OGPQk_Yl1iGX4-eCHWLgE/edit?usp=sharing.  I have significantly reduced the content in the Flip, in line with what I see as the template and its usage. If this it too much or too little, I can amend,

    Kind regards, David.

From: Martijn Visser <ma...@apache.org>
Date: Friday, 12 April 2024 at 18:11
To: dev@flink.apache.org <de...@flink.apache.org>
Subject: Re: [EXTERNAL] RE: [DISCUSS] FLIP-XXX Apicurio-avro format
Hi David,

I tried, but the format wasn't as the FLIP template expects, so I ended up
needing to change the entire formatting and that was just too much work to
be honest. If you could make sure that especially the headers match with
the FLIP template, and that all of the contents from the FLIP template is
there, that would make things much easier.

Thanks,

Martijn

On Fri, Apr 12, 2024 at 6:08 PM David Radley <da...@uk.ibm.com>
wrote:

> Hi,
> A gentle nudge. Please could a committer/PMC member raise the Flip for
> this,
>       Kind regards, David.
>
>
> From: David Radley <da...@uk.ibm.com>
> Date: Monday, 8 April 2024 at 09:40
> To: dev@flink.apache.org <de...@flink.apache.org>
> Subject: [EXTERNAL] RE: [DISCUSS] FLIP-XXX Apicurio-avro format
> Hi,
> I have posted a Google Doc [0] to the mailing list for a discussion thread
> for a Flip proposal to introduce a Apicurio-avro format. The discussions
> have been resolved, please could a committer/PMC member copy the contents
> from the Google Doc, and create a FLIP number for this,. as per the process
> [1],
>       Kind regards, David.
> [0]
>
> https://docs.google.com/document/d/14LWZPVFQ7F9mryJPdKXb4l32n7B0iWYkcOdEd1xTC7w/edit?usp=sharing
>
> [1]
>
> https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals#FlinkImprovementProposals-CreateyourOwnFLIP
>
> From: Jeyhun Karimov <je...@gmail.com>
> Date: Friday, 22 March 2024 at 13:05
> To: dev@flink.apache.org <de...@flink.apache.org>
> Subject: [EXTERNAL] Re: [DISCUSS] FLIP-XXX Apicurio-avro format
> Hi David,
>
> Thanks a lot for clarification.
> Sounds good to me.
>
> Regards,
> Jeyhun
>
> On Fri, Mar 22, 2024 at 10:54 AM David Radley <da...@uk.ibm.com>
> wrote:
>
> > Hi Jeyhun,
> > Thanks for your feedback.
> >
> > So for outbound messages, the message includes the global ID. We register
> > the schema and match on the artifact id. So if the schema then evolved,
> > adding a new  version, the global ID would still be unique and the same
> > version would be targeted. If you wanted to change the Flink table
> > definition in line with a higher version, then you could do this – the
> > artifact id would need to match for it to use the same schema and a
> higher
> > artifact version would need to be provided. I notice that Apicurio has
> > rules around compatibility that you can configure, I suppose if we
> attempt
> > to create an artifact that breaks these rules , then the register schema
> > will fail and the associated operation should fail (e.g. an insert). I
> have
> > not tried this.
> >
> >
> > For inbound messages, using the global id in the header – this targets
> one
> > version of the schema. I can create different messages on the topic built
> > with different schema versions, and I can create different tables in
> Flink,
> > as long as the reader and writer schemas are compatible as per the
> >
> https://github.com/apache/flink/blob/779459168c46b7b4c600ef52f99a5435f81b9048/flink-formats/flink-avro/src/main/java/org/apache/flink/formats/avro/RegistryAvroDeserializationSchema.java#L109
> > Then this should work.
> >
> > Does this address your question?
> >     Kind regards, David.
> >
> >
> > From: Jeyhun Karimov <je...@gmail.com>
> > Date: Thursday, 21 March 2024 at 21:06
> > To: dev@flink.apache.org <de...@flink.apache.org>
> > Subject: [EXTERNAL] Re: [DISCUSS] FLIP-XXX Apicurio-avro format
> > Hi David,
> >
> > Thanks for the FLIP. +1 for it.
> > I have a minor comment.
> >
> > Can you please elaborate more on mechanisms in place to ensure data
> > consistency and integrity, particularly in the event of schema conflicts?
> > Since each message includes a schema ID for inbound and outbound
> messages,
> > can you elaborate more on message consistency in the context of schema
> > evolution?
> >
> > Regards,
> > Jeyhun
> >
> >
> >
> >
> >
> > On Wed, Mar 20, 2024 at 4:34 PM David Radley <da...@apache.org>
> wrote:
> >
> > > Thank you very much for your feedback Mark. I have made the changes in
> > the
> > > latest google document. On reflection I agree with you that the
> > > globalIdPlacement format configuration should apply to the
> > deserialization
> > > as well, so it is declarative. I am also going to have a new
> > configuration
> > > option to work with content IDs as well as global IDs. In line with the
> > > deser Apicurio IdHandler and headerHandlers.
> > >
> > >  kind regards, David.
> > >
> > >
> > > On 2024/03/20 15:18:37 Mark Nuttall wrote:
> > > > +1 to this
> > > >
> > > > A few small comments:
> > > >
> > > > Currently, if users have Avro schemas in an Apicurio Registry (an
> open
> > > source Apache 2 licensed schema registry), then the natural way to work
> > > with those Avro flows is to use the schemas in the Apicurio Repository.
> > > > 'those Avro flows' ... this is the first reference to flows.
> > > >
> > > > The new format will use the global Id to look up the Avro schema that
> > > the message was written during deserialization.
> > > > I get the point, phrasing is awkward. Probably you're more interested
> > in
> > > content than word polish at this point though.
> > > >
> > > > The Avro Schema Registry (apicurio-avro) format
> > > > The Confluent format is called avro-confluent; this should be
> > > avro-apicurio
> > > >
> > > > How to create tables with Apicurio-avro format
> > > > s/Apicurio-avro/avro-apicurio/g
> > > >
> > > > HEADER – globalId is put in the header
> > > > LEGACY– global Id is put in the message as a long
> > > > CONFLUENT - globalId is put in the message as an int.
> > > > Please could we specify 'four-byte int' and 'eight-byte long' ?
> > > >
> > > > For a Kafka source the globalId will be looked for in this order:
> > > > -     In the header
> > > > -     After a magic byte as an int
> > > > -     After a magic byte as a long.
> > > > but apicurio-avro.globalid-placement has a default value of HEADER :
> > why
> > > do we have a search order as well? Isn't
> apicurio-avro.globalid-placement
> > > enough? Don't the two mechanisms conflict?
> > > >
> > > > In addition to the types listed there, Flink supports reading/writing
> > > nullable types. Flink maps nullable types to Avro union(something,
> null),
> > > where something is the Avro type converted from Flink type.
> > > > Is that definitely the right way round? I know we've had multiple
> > > conversations about how unions work with Flink
> > > >
> > > >  This is because the writer schema is expanded, but this could not
> > > complete if there are circularities.
> > > > I understand your meaning but the sentence is awkward.
> > > >
> > > > The registered schema will be created or if it exists be updated.
> > > > same again
> > > >
> > > > At some stage the lowest Flink level supported by the Kafka connector
> > > will contain the additionalProperties methods in code flink.
> > > > wording
> > > >
> > > > There existing Kafka deserialization for the writer schema passes
> down
> > > the message body to be deserialised.
> > > > wording
> > > >
> > > > @Override
> > > > public void deserialize(ConsumerRecord<byte[], byte[]> message,
> > > Collector<T> out)
> > > >       throws IOException {
> > > >       Map<String, Object> additionalPropertiesMap =  new HashMap<>();
> > > >       for (Header header : message.additionalProperties()) {
> > > >       headersMap.put(header.key(), header.value());
> > > >       }
> > > >       deserializationSchema.deserialize(message.value(), headersMap,
> > > out);
> > > > }
> > > > This fails to compile at headersMap.
> > > >
> > > > The input stream and additionalProperties will be sent so the
> Apicurio
> > > SchemaCoder which will try getting the globalId from the headers, then
> 4
> > > bytes from the payload then 8 bytes from the payload.
> > > > I'm still stuck on apicurio-avro.globalid-placement having a default
> > > value of HEADER . Should we try all three, or fail if this config param
> > has
> > > a wrong value?
> > > >
> > > > Other considerations
> > > > The implementation does not use the Apicurio deser libraries,
> > > > Please can we refer to them as SerDes; this is the term used within
> the
> > > documentation that you link to
> > > >
> > > >
> > > > On 2024/03/20 10:09:08 David Radley wrote:
> > > > > Hi,
> > > > > As per the FLIP process I would like to raise a FLIP, but do not
> have
> > > authority, so have created a google doc for the Flip to introduce a new
> > > Apicurio Avro format. The document is
> > >
> >
> https://docs.google.com/document/d/14LWZPVFQ7F9mryJPdKXb4l32n7B0iWYkcOdEd1xTC7w/edit?usp=sharing
> > > > >
> > > > > I have prototyped a lot of the content to prove that this approach
> is
> > > feasible. I look forward to the discussion,
> > > > >       Kind regards, David.
> > > > >
> > > > >
> > > > >
> > > > > Unless otherwise stated above:
> > > > >
> > > > > IBM United Kingdom Limited
> > > > > Registered in England and Wales with number 741598
> > > > > Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6
> > 3AU
> > > > >
> > > >
> > >
> >
> > Unless otherwise stated above:
> >
> > IBM United Kingdom Limited
> > Registered in England and Wales with number 741598
> > Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU
> >
>
> Unless otherwise stated above:
>
> IBM United Kingdom Limited
> Registered in England and Wales with number 741598
> Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU
>

Unless otherwise stated above:

IBM United Kingdom Limited
Registered in England and Wales with number 741598
Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU

FW: RE: [DISCUSS] FLIP-XXX Apicurio-avro format

Posted by David Radley <da...@uk.ibm.com>.
Hi Arvid,
I realise that the reason I did not follow the approach you suggested, is that the config I had in the Kafka connector was that of the connector (not the format)  – so I could not get hold of the format name to do a lookup of a factory based on the format name. I will do some more testing to confirm my understanding either way. If we can get the format name in the connector – I agree your proposed solution is neater,
     Warm regards, David.



From: David Radley <da...@uk.ibm.com>
Date: Tuesday, 24 September 2024 at 22:16
To: dev@flink.apache.org <de...@flink.apache.org>
Subject: [EXTERNAL] RE: [DISCUSS] FLIP-XXX Apicurio-avro format
Hi Arvid,
Thank you very much for looking at this and for Robert to pass this through to you. Some quick thoughts.

The current design is not because it came from an earlier version of the fix, Chesnay and I came up with this  design, is because we took the assumptions that:

  *   the existing code serialise and deserialise methods that the Kafka connector is using.
  *   We wanted to use the existing Avro format – so do the Avro part of the serialization / deserialization
  *   The reasons we need the record is because the schema information is in the headers by default for Apicurio.

Happy to step back on this.
I was thinking if we want to change these assumptions, then the impact of the change would be bigger as we would need to reimagine the Kafka or Avro relationship with this format. Your idea of a new format factory for the deserialization and serialization, also sounds good.

On your solutions :
Re 1- this is what we have.
Re 2 and 3 I will investigate your suggestions. I like the sound of them and at first glance I don’t see anything obvious that you have not considered.

  *   If we can have a format specific factory that passes down the record for the payload for Apicurio. I just need to check that the metadata processing still works as expected.
  *   I assume that the default in the Factory will be to do what it currently does then for Apicurio we pass down the record. In this way it should not effect Confluent Avro.
  *   I assume that the connector knows about the format when it needs to do the serialization / deserialization – I need to check the code on this.
  *   I think the fix would basically be the same apart from the bespoke serialization – it would just pass down the record. And the format factory instead of class loading the new serialization / deserialization.
  *   I have not hardwired it into the Kafka connector. If we have the apicurio hard wired into the Kafka connector jar then those classes will be there when using Confluent Avro. I suggest we keep the format as a separate format jar, and SQL jar like the other formats.

On the if you want to be fancy – a variant of this was my first proposed fix 😊 .
   Kind regards, David.

From: Arvid Heise <ar...@apache.org>
Date: Tuesday, 24 September 2024 at 21:11
To: Robert Metzger <rm...@apache.org>
Cc: dev@flink.apache.org <de...@flink.apache.org>
Subject: [EXTERNAL] Re: [DISCUSS] FLIP-XXX Apicurio-avro format
Hi Robert, thanks for looping me in.

I have looked at the branch and the FLIP. Apicurio looks like a promising
alternative to Confluent SR and I'm certain that it's a good addition to
Flink.

However, at the current form it looks heavily overengineered. I'm
suspecting that comes from the attempt to do some generic implementation in
flink-core and later moving it into the kafka connector. I'm concerned for
two reasons: there is lots of code involved for a relatively minor change
that needs to be maintained and it looks like performance is going to
suffer from using SPI on the hotpath. I also don't fully understand why we
need to decide per record on which deserializer is the most suitable; I'd
expect one topic to either use Apicurio or not.

My proposal is to take a step back and ask what is really needed and
desired. I think from a high level, we need
1. a way for the user to specify that it's Apicurio and provide all the
config options
2. we need to modify the KafkaDynamicSource/Sink to use the respective
Apicurio (de)serializer
3. we need to implement the (de)serializer

Let's look at solutions
Re 1: This is straight-forward. I haven't seen this explicitly in the FLIP
and it might be worthwhile to call it out. It probably looks like this

CREATE TABLE user_created (
 ...
) WITH (

 'connector' = 'kafka',
 'topic' = 'user',
 'properties.bootstrap.servers' = '<broker>:9092',

 'key.format' = 'avro-apicurio',
 'key.avro-apicurio.url' = 'http://<apicurio>:8082<http://%3capicurio%3e:8082<http://%3capicurio%3e:8082%3chttp:/%3capicurio%3e:8082>>',
 ...

 'value.format' = 'avro-apicurio',
 'value.avro-apicurio.url' = 'http://<apicurio>:8082<http://%3capicurio%3e:8082<http://%3capicurio%3e:8082%3chttp:/%3capicurio%3e:8082>>',
 ...
)

Re 2: This is the hardest part as the KafkaDynamicSource/Sink basically
uses (De)serializationSchema which just knows bytes and can't use
ConsumerRecord. I see how you came to some conclusions.
However, I think there is still some easier way to achieve what you want:
We need to use the generified KafkaRecord(De)serializationSchema instead.
Let's check what needs to be done for KafkaDynamicSource (sink should be
similar):
* Replace all DeserializationSchema with KafkaRecordDeserializationSchema
in KafkaDynamicSource and DynamicKafkaDeserializationSchema
* Introduce a KafkaRecordDeserializationFormatFactory with an
implementation that delegates to existing factory and
uses KafkaRecordDeserializationSchema#valueOnly under the hood.
* Look in KafkaDynamicTableFactory for the old factory and wrap it into the
new factory. Optionally also look for the new factory (see below).

Re 3:
Implement Apicurio using KafkaRecordDeserializationSchema. This would be
directly usable in DataStream API.
For table API, you can implement the new
KafkaRecordDeserializationFormatFactory and dynamically find it. However,
as of now it seems like everything is in the same jar already. If it's
never really separated, I'm also fine to hardwire the Apicurio
implementations inside the KafkaDynamicTableFactory (so simply add it
statically to the SPI formats).

So as you can see, there is no need to introduce any kind of new SPI lookup
or did I miss something obvious?



Btw if we want to be fancy, an alternative/extended solution would be to
generify (De)serializationSchema in flink-core. A
RecordDeserializationSchema<R, T> would have a

void deserialize(R record, Collector<T> out) throws IOException;

and then you could plug in different kinds of records.
The existing DeserializationSchema would become

public interface DeserializationSchema<T> extends
RecordDeserializationSchema<byte[], T>

The KafkaRecordDeserializationSchema

public interface KafkaRecordDeserializationSchema<T> extends
RecordDeserializationSchema<ConsumerRecord<byte[], byte[]>, T>

However, that's a FLIP on its own, especially to maintain backwards
compatibility.

Best.

Arvid

PS: don't hesitate to reach out on slack for quicker feedback. we should
still primarily use the ML to keep everyone in the loop but it's usually
much faster to ping me there if you post an answer here.

On Tue, Sep 24, 2024 at 7:50 PM Robert Metzger <rm...@apache.org> wrote:

> @Arvid: Could you take a look at this.
>
> Thanks a lot.
>
> On Wed, Aug 14, 2024 at 6:19 PM David Radley <da...@uk.ibm.com>
> wrote:
>
>> Hi Danny,
>> Thank you for your feedback. I have brought a fork up to date with one
>> commit so you can
>> see the code; I need to test more scenarios and add more unit tests, but
>> it shows the basic idea of the design, Chesnay and I agreed.
>>
>>  Kind regards, David.
>>
>>
>> From: Danny Cranmer <da...@apache.org>
>> Date: Thursday, 25 July 2024 at 16:46
>> To: dev@flink.apache.org <de...@flink.apache.org>
>> Cc: David Radley <da...@uk.ibm.com>
>> Subject: [EXTERNAL] Re: [DISCUSS] FLIP-XXX Apicurio-avro format
>> Hey, Thanks for the updated FLIP. > 1. 20 or beyond Given this is
>> intended to go in the Kafka connector we can target the next Kafka
>> connector version (v3. 3. 0) for Flink 1. 18 and 1. 19 (or 1. 19 and 1. 20,
>> if 1. 20 is released first). > RecordBasedSerialization: : public
>>
>> Hey,
>>
>> Thanks for the updated FLIP.
>>
>> > 1.20 or beyond
>>
>> Given this is intended to go in the Kafka connector we can target the
>> next Kafka connector version (v3.3.0) for Flink 1.18 and 1.19 (or 1.19 and
>> 1.20, if 1.20 is released first).
>>
>>
>> >> Agreed
>>
>>  > RecordBasedSerialization::public boolean canProcess(byte[]
>> customSerialisation);
>>
>> Not sure what this is for, since we are passing a byte array and asking
>> if it can be serialized? Can you elaborate how the Kafka connector
>> interacts with these interfaces?
>> >> we are asking it can be processed. The canProcess has a default and
>> also can be overridden. The idea is that if we are in another format and
>> the apicurio jar is there then we might pick up the apicurio jar in the
>> discovery. So we check that what we have discovered really can process the
>> message. During trsting I have the key as raw and the value as apicurio,
>> and saw the issue that necessitates this method.
>>
>>
>> > byte[] customSerialisation
>>
>> There are a lot of byte arrays being passed around, I assume we are not
>> doing heavy conversions multiple times? Is there scope to introduce a new
>> intermediate type to prevent duplicate processing cycles?
>> Or even better, do you have a fork with these changes you can share?
>> >> https://github.com/davidradl/flink-connector-kafka/tree/FLINK-35311-3
>>
>> > Format 'avro-apicurio'
>>
>> Since this is Kafka only I would be tempted to include Kafka in this tag,
>> just in case we decide to add a Kafka agnostic format in the future.
>> Additionally this makes it clearer to the user it is Kafka only.
>> >> I was keeping this in line with the avro-confluent naming which is
>> also Kafka only. I do not have strong views on this – so I can change if we
>> think it is better.
>>
>> Thanks,
>> Danny
>>
>>
>>
>> On Thu, Jul 18, 2024 at 6:32 PM Martijn Visser <martijnvisser@apache.org
>> <ma...@apache.org>> wrote:
>> Hi David,
>>
>> The FLIP is updated.
>>
>> Cheers, Martijn
>>
>>
>> On Tue, Jul 16, 2024 at 6:33 PM David Radley <david_radley@uk.ibm.com
>> <ma...@uk.ibm.com>>
>> wrote:
>>
>> > Hello all,
>> >
>> >
>> >
>> > I have prototyped the new design and confirmed it works. I have
>> documented
>> > the design in google doc
>> >
>> https://docs.google.com/document/d/1J1E-cE-X2H3-kw4rNjLn71OGPQk_Yl1iGX4-eCHWLgE/edit
>> >
>> >
>> >
>> > By copy Martijn: please could you move this content over to replace the
>> > content of
>> >
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-454%3A+New+Apicurio+Avro+format
>> >
>> >
>> >
>> > As the design is significantly different, there will be a new period of
>> > discussion before going to vote.
>> >
>> >
>> >
>> > Fyi I will be on vacation after this week until the 13th of August.
>> >
>> >
>> >
>> > Kind regards, David.
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> > *From: *David Radley <david_radley@uk.ibm.com<mailto:
>> david_radley@uk.ibm.com>>
>> > *Date: *Tuesday, 9 July 2024 at 11:17
>> > *To: *dev@flink.apache.org<ma...@flink.apache.org> <
>> dev@flink.apache.org<ma...@flink.apache.org>>
>> > *Subject: *[EXTERNAL] RE: [DISCUSS] FLIP-XXX Apicurio-avro format
>> >
>> > Hi Kevin,
>> > I have agreed a design with Chesnay and Danny. I am implementing a
>> > prototype, to prove it works,  then will update the Flip text with the
>> new
>> > design. Initial testing is showing it working.
>> >
>> > Here is a quick history so you can understand our current thinking.
>> >
>> >   1.  Initially we passed maps for header information from the kafka
>> > connector to Flink for deserialization. Similar for serialize. This was
>> not
>> > great, because maps are not ideal and it was a big change as it needed
>> core
>> > Flink interface changes
>> >   2.  We then moved the Avro Apicurio format to the Kafka connector and
>> > looked to discover a new record based de/serialization interface. So we
>> > could pass down the record (containing the headers) rather than the
>> > payload. This did not work, because there is a dependence on the Avro
>> > connector that is not  aware of the new interface.
>> >   3.  We considered using Thread local storage to pass the headers, we
>> did
>> > not like this as there was a risk of memory leaks if we did not manage
>> the
>> > thread well, also the contract is hidden.
>> >   4.  We then came up with the current design that augments the
>> > deserialization in the Kafka connector in a new discovered record based
>> > deserialization, it then takes the headers out in the schema coder,
>> leaving
>> > the message as it was. Similar for serialization.
>> >
>> >
>> > One piece I need to work out the details of, is how to work when there
>> are
>> > 2 implementations that can be discovered, probably using an augmented
>> > format name as a factory identifier,
>> >
>> > I hope to put up a new design in the Flip by the end of next week, for
>> > wider review,
>> >     Kind regards, David.
>> >
>> >
>> > From: Kevin Lam <ke...@shopify.com.INVALID>
>> > Date: Monday, 8 July 2024 at 21:16
>> > To: dev@flink.apache.org<ma...@flink.apache.org> <
>> dev@flink.apache.org<ma...@flink.apache.org>>
>> > Subject: [EXTERNAL] Re: [DISCUSS] FLIP-XXX Apicurio-avro format
>> > Hi David,
>> >
>> > Any updates on the Kafka Message Header support? I am also interested in
>> > supporting headers with the Flink SQL Formats:
>> > https://lists.apache.org/thread/spl88o63sjm2dv4l5no0ym632d2yt2o6
>> >
>> > On Fri, Jun 14, 2024 at 6:10 AM David Radley <david_radley@uk.ibm.com
>> <ma...@uk.ibm.com>>
>> > wrote:
>> >
>> > > Hi everyone,
>> > > I have talked with Chesnay and Danny offline. Danny and I were not
>> very
>> > > happy with the passing Maps around, and were looking for a neater
>> design.
>> > > Chesnay suggested that we could move the new format to the Kafka
>> > connector,
>> > > then pass the Kafka record down to the deserialize logic so it can
>> make
>> > use
>> > > of the headers during deserialization and serialisation.
>> > >
>> > > I think this is a neat idea. This would mean:
>> > > - the Kafka connector code would need to be updated to pass down the
>> > Kafka
>> > > record
>> > > - there would be the Avro Apicurio format and SQL in the kafka
>> > repository.
>> > > We feel it is unlikely to want to use the Apicurio registry with
>> files,
>> > as
>> > > the Avro format could be used.
>> > >
>> > > Unfortunately I have found that this as not so straight forward to
>> > > implement as the Avro Apicurio format uses the Avro format, which is
>> tied
>> > > to the DeserializationSchema. We were hoping to have a new decoding
>> > > implementation that would pass down the Kafka record rather than the
>> > > payload. This does not appear possible without a Avro format change.
>> > >
>> > >
>> > > Inspired by this idea, I notice that
>> > > KafkaValueOnlyRecordDeserializerWrapper<T> extends
>> > > KafkaValueOnlyDeserializerWrapper
>> > >
>> > > Does
>> > >
>> > > deserializer.deserialize(record.topic(),record.value())
>> > >
>> > >
>> > >
>> > > I am investigating If I can add a factory/reflection to provide an
>> > > alternative
>> > > Implementation that will pass the record based (the kafka record is
>> not
>> > > serializable so I will pick what we need and deserialize) as a byte
>> > array.
>> > >
>> > > I would need to do this 4 times (value ,key for deserialisation and
>> > > serialisation. To do this I would need to convert the record into a
>> byte
>> > > array, so it fits into the existing interface
>> (DeserializationSchema).  I
>> > > think this could be a way through, to avoid using maps and avoid
>> changing
>> > > the existing Avro format and avoid change any core Flink interfaces.
>> > >
>> > > I am going to prototype this idea. WDYT?
>> > >
>> > > My thanks go to Chesnay and Danny for their support and insight around
>> > > this Flip,
>> > >    Kind regards, David.
>> > >
>> > >
>> > >
>> > >
>> > >
>> > >
>> > > From: David Radley <david_radley@uk.ibm.com<mailto:
>> david_radley@uk.ibm.com>>
>> > > Date: Wednesday, 29 May 2024 at 11:39
>> > > To: dev@flink.apache.org<ma...@flink.apache.org> <
>> dev@flink.apache.org<ma...@flink.apache.org>>
>> > > Subject: [EXTERNAL] RE: [DISCUSS] FLIP-XXX Apicurio-avro format
>> > > Hi Danny,
>> > > Thank you for your feedback on this.
>> > >
>> > > I agree that using maps has pros and cons. The maps are flexible, but
>> do
>> > > require the sender and receiver to know what is in the map.
>> > >
>> > > When you say “That sounds like it would fit in better, I assume we
>> cannot
>> > > just take that approach?” The motivation behind this Flip is to
>> support
>> > the
>> > > headers which is the usual way that Apicurio runs. We will support the
>> > > “schema id in the payload” as well.
>> > >
>> > > I agree with you when you say “ I am not 100% happy with the solution
>> > but I
>> > > cannot offer a better option.” – this is a pragmatic way we have
>> found to
>> > > solve this issue. I am open to any suggestions to improve this as
>> well.
>> > >
>> > > If we are going with the maps design (which is the best we have at the
>> > > moment) ; it would be good to have the Flink core changes in base
>> Flink
>> > > version 2.0 as this would mean we do not need to use reflection in a
>> > Flink
>> > > Kafka version 2 connector to work out if the runtime Flink has the new
>> > > methods.
>> > >
>> > > At this stage we only have one committer (yourself) backing this. Do
>> you
>> > > know of other 2 committers who would support this Flip?
>> > >
>> > >      Kind regards, David.
>> > >
>> > >
>> > >
>> > > From: Danny Cranmer <dannycranmer@apache.org<mailto:
>> dannycranmer@apache.org>>
>> > > Date: Friday, 24 May 2024 at 19:32
>> > > To: dev@flink.apache.org<ma...@flink.apache.org> <
>> dev@flink.apache.org<ma...@flink.apache.org>>
>> > > Subject: [EXTERNAL] Re: [DISCUSS] FLIP-XXX Apicurio-avro format
>> > > Hello,
>> > >
>> > > > I am curious what you mean by abused.
>> > >
>> > > I just meant we will end up adding more and more fields to this map
>> over
>> > > time, and it may be hard to undo.
>> > >
>> > > > For Apicurio it can be sent at the start of the payload like
>> Confluent
>> > > Avro does. Confluent Avro have a magic byte followed by 4 bytes of
>> schema
>> > > id, at the start of the payload. Apicurio clients and SerDe libraries
>> can
>> > > be configured to not put the schema id in the headers in which case
>> there
>> > > is a magic byte followed by an 8 byte schema at the start of the
>> payload.
>> > > In the deserialization case, we would not need to look at the headers
>> –
>> > > though the encoding is also in the headers.
>> > >
>> > > That sounds like it would fit in better, I assume we cannot just take
>> > that
>> > > approach?
>> > >
>> > > Thanks for the discussion. I am not 100% happy with the solution but I
>> > > cannot offer a better option. I would be interested to hear if others
>> > have
>> > > any suggestions. Playing devil's advocate against myself, we pass maps
>> > > around to configure connectors so it is not too far away from that.
>> > >
>> > > Thanks,
>> > > Danny
>> > >
>> > >
>> > > On Fri, May 24, 2024 at 2:23 PM David Radley <david_radley@uk.ibm.com
>> <ma...@uk.ibm.com>>
>> > > wrote:
>> > >
>> > > > Hi Danny,
>> > > > No worries, thanks for replying. I have working prototype code that
>> is
>> > > > being reviewed. It needs some cleaning up and more complete testing
>> > > before
>> > > > it is ready, but will give you the general idea [1][2] to help to
>> > assess
>> > > > this approach.
>> > > >
>> > > >
>> > > > I am curious what you mean by abused. I guess the approaches are
>> > between
>> > > > generic map, mechanism vs a more particular more granular things
>> being
>> > > > passed that might be used by another connector.
>> > > >
>> > > > Your first question:
>> > > > “how would this work if the schema ID is not in the Kafka headers,
>> as
>> > > > hinted to in the FLIP "usually the global ID in a Kafka header"?
>> > > >
>> > > > For Apicurio it can be sent at the start of the payload like
>> Confluent
>> > > > Avro does. Confluent Avro have a magic byte followed by 4 bytes of
>> > schema
>> > > > id, at the start of the payload. Apicurio clients and SerDe
>> libraries
>> > can
>> > > > be configured to not put the schema id in the headers in which case
>> > there
>> > > > is a magic byte followed by an 8 byte schema at the start of the
>> > payload.
>> > > > In the deserialization case, we would not need to look at the
>> headers –
>> > > > though the encoding is also in the headers.
>> > > >
>> > > > Your second question:
>> > > > “I am wondering if there are any other instances where the source
>> would
>> > > be
>> > > > aware of the schema ID and pass it through in this way?
>> > > > ”
>> > > > The examples I can think of are:
>> > > > - Avro can send the complete schema in a header, this is not
>> > recommended
>> > > > but in theory fits the need for a message payload to require
>> something
>> > > else
>> > > > to get the structure.
>> > > > - I see [2] that Apicurio Protobuf uses headers.
>> > > > - it might be that other message queuing projects like Rabbit MQ
>> would
>> > > > need this to be able to support Apicurio Avro & protobuf.
>> > > >
>> > > > Kind regards, David,
>> > > >
>> > > >
>> > > >
>> > > >
>> > > > [1] https://github.com/apache/flink/pull/24715
>> > > > [2] https://github.com/apache/flink-connector-kafka/pull/99
>> > > > [3]
>> > > >
>> > >
>> >
>> https://www.apicur.io/registry/docs/apicurio-registry/2.5.x/getting-started/assembly-configuring-kafka-client-serdes.html#registry-serdes-types-json_registry
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > > From: Danny Cranmer <dannycranmer@apache.org<mailto:
>> dannycranmer@apache.org>>
>> > > > Date: Friday, 24 May 2024 at 12:22
>> > > > To: dev@flink.apache.org<ma...@flink.apache.org> <
>> dev@flink.apache.org<ma...@flink.apache.org>>
>> > > > Subject: [EXTERNAL] Re: [DISCUSS] FLIP-XXX Apicurio-avro format
>> > > > Hello,
>> > > >
>> > > > Apologies, I am on vacation and have limited access to email.
>> > > >
>> > > > I can see the logic here and why you ended up where you did. I can
>> also
>> > > see
>> > > > there are other useful metadata fields that we might want to pass
>> > > through,
>> > > > which might result in this Map being abused (Kafka Topic, Kinesis
>> > Shard,
>> > > > etc).
>> > > >
>> > > > I have a follow up question, how would this work if the schema ID is
>> > not
>> > > in
>> > > > the Kafka headers, as hinted to in the FLIP "usually the global ID
>> in a
>> > > > Kafka header"? I am wondering if there are any other instances where
>> > the
>> > > > source would be aware of the schema ID and pass it through in this
>> way?
>> > > >
>> > > > Thanks,
>> > > > Danny
>> > > >
>> > > >
>> > > >
>> > > > On Wed, May 22, 2024 at 3:43 PM David Radley <
>> david_radley@uk.ibm.com<ma...@uk.ibm.com>>
>> > > > wrote:
>> > > >
>> > > > > Hi Danny,
>> > > > > Did you have a chance you have a look at my responses to your
>> > > feedback? I
>> > > > > am hoping to keep the momentum going on this one,   kind regards,
>> > > David.
>> > > > >
>> > > > >
>> > > > > From: David Radley <david_radley@uk.ibm.com<mailto:
>> david_radley@uk.ibm.com>>
>> > > > > Date: Tuesday, 14 May 2024 at 17:21
>> > > > > To: dev@flink.apache.org<ma...@flink.apache.org> <
>> dev@flink.apache.org<ma...@flink.apache.org>>
>> > > > > Subject: [EXTERNAL] [DISCUSS] FLIP-XXX Apicurio-avro format
>> > > > > Hi Danny,
>> > > > >
>> > > > > Thank you very much for the feedback and your support. I have
>> copied
>> > > your
>> > > > > feedback from the VOTE thread to this discussion thread, so we can
>> > > > continue
>> > > > > our discussions off the VOTE thread.
>> > > > >
>> > > > >
>> > > > >
>> > > > > Your feedback:
>> > > > >
>> > > > > Thanks for Driving this David. I am +1 for adding support for the
>> new
>> > > > >
>> > > > > format, however have some questions/suggestions on the details.
>> > > > >
>> > > > >
>> > > > >
>> > > > > 1. Passing around Map<String, Object> additionalInputProperties
>> > feels a
>> > > > bit
>> > > > >
>> > > > > dirty. It looks like this is mainly for the Kafka connector. This
>> > > > connector
>> > > > >
>> > > > > already has a de/serialization schema extension to access record
>> > > > >
>> > > > > headers, KafkaRecordDeserializationSchema [1], can we use this
>> > instead?
>> > > > >
>> > > > > 2. Can you elaborate why we need to change the SchemaCoder
>> interface?
>> > > > Again
>> > > > >
>> > > > > I am not a fan of adding these Map parameters
>> > > > >
>> > > > > 3. I assume this integration will go into the core Flink repo
>> under
>> > > > >
>> > > > > flink-formats [2], and not be a separate repository like the
>> > > connectors?
>> > > > >
>> > > > >
>> > > > >
>> > > > > My response:
>> > > > >
>> > > > > Addressing 1. and 2.
>> > > > >
>> > > > > I agree that sending maps around is a bit dirty. If we can see a
>> > better
>> > > > > way that would be great. I was looking for a way to pass this
>> kafka
>> > > > header
>> > > > > information in a non-Kafka way - the most obvious way I could
>> think
>> > was
>> > > > as
>> > > > > a map. Here are the main considerations I saw, if I have missed
>> > > anything
>> > > > or
>> > > > > could improve something I would be grateful for any further
>> feedback.
>> > > > >
>> > > > >
>> > > > >
>> > > > >   *   I see KafkaRecordDeserializationSchema is a Kafka interface
>> > that
>> > > > > works at the Kafka record level (so includes the headers). We
>> need a
>> > > > > mechanism to send over the headers from the Kafka record to Flink
>> > > > >   *   Flink core is not aware of Kafka headers, and I did not
>> want to
>> > > add
>> > > > > a Kafka dependancy to core flink.
>> > > > >   *   The formats are stateless so it did not appear to be in
>> fitting
>> > > > with
>> > > > > the Flink architecture to pass through header information to
>> stash in
>> > > > state
>> > > > > in the format waiting for the deserialise to be subsequently
>> called
>> > to
>> > > > pick
>> > > > > up the header information.
>> > > > >   *   We could have used Thread local storage to stash the header
>> > > > content,
>> > > > > but this would be extra state to manage; and this would seem like
>> an
>> > > > > obtrusive change.
>> > > > >   *   The SchemaCoder deserialise is where Confluent Avro gets the
>> > > schema
>> > > > > id from the payload, so it can lookup the schema. In line with
>> this
>> > > > > approach it made sense to extend the deserialise so it had the
>> header
>> > > > > contents so the Apicurio Avro format could lookup the schema.
>> > > > >   *   I did not want to have Apicurio specific logic in the Kafka
>> > > > > connector, if we did we could pull out the appropriate headers and
>> > only
>> > > > > send over the schema ids.
>> > > > >   *   For deserialise, the schema id we are interested in is the
>> one
>> > in
>> > > > > the Kafka headers on the message and is for the writer schema (an
>> > Avro
>> > > > > format concept) currently used by the confluent-avro format in
>> > > > deserialize.
>> > > > >   *   For serialise the schema ids need to be obtained from
>> apicurio
>> > > then
>> > > > > passed through to Kafka.
>> > > > >   *   For serialise there is existing logic around handling the
>> > > metadata
>> > > > > which includes passing the headers. But the presence of the
>> metadata
>> > > > would
>> > > > > imply we have a metadata column. Maybe a change to the metadata
>> > > mechanism
>> > > > > may have allowed to use to pass the headers, but not create a
>> > metadata
>> > > > > column; instead I pass through the additional headers in a map to
>> be
>> > > > > appended.
>> > > > >
>> > > > >
>> > > > >
>> > > > > 3.
>> > > > >
>> > > > > Yes this integration will go into the core Flink repo under
>> > > > >
>> > > > > flink-formats and sit next to the confluent-avro format. The Avro
>> > > format
>> > > > > has the concept of a Registry and drives the confluent-avro
>> format.
>> > The
>> > > > > Apicurio Avro format will use the same approach.
>> > > > >
>> > > > > Unless otherwise stated above:
>> > > > >
>> > > > > IBM United Kingdom Limited
>> > > > > Registered in England and Wales with number 741598
>> > > > > Registered office: PO Box 41, North Harbour, Portsmouth, Hants.
>> PO6
>> > 3AU
>> > > > >
>> > > > > Unless otherwise stated above:
>> > > > >
>> > > > > IBM United Kingdom Limited
>> > > > > Registered in England and Wales with number 741598
>> > > > > Registered office: PO Box 41, North Harbour, Portsmouth, Hants.
>> PO6
>> > 3AU
>> > > > >
>> > > >
>> > > > Unless otherwise stated above:
>> > > >
>> > > > IBM United Kingdom Limited
>> > > > Registered in England and Wales with number 741598
>> > > > Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6
>> 3AU
>> > > >
>> > >
>> > > Unless otherwise stated above:
>> > >
>> > > IBM United Kingdom Limited
>> > > Registered in England and Wales with number 741598
>> > > Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6
>> 3AU
>> > >
>> > > Unless otherwise stated above:
>> > >
>> > > IBM United Kingdom Limited
>> > > Registered in England and Wales with number 741598
>> > > Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6
>> 3AU
>> > >
>> >
>> > Unless otherwise stated above:
>> >
>> > IBM United Kingdom Limited
>> > Registered in England and Wales with number 741598
>> > Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU
>> > Unless otherwise stated above:
>> >
>> > IBM United Kingdom Limited
>> > Registered in England and Wales with number 741598
>> > Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU
>> >
>>
>> Unless otherwise stated above:
>>
>> IBM United Kingdom Limited
>> Registered in England and Wales with number 741598
>> Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU
>>
>

Unless otherwise stated above:

IBM United Kingdom Limited
Registered in England and Wales with number 741598
Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU

Unless otherwise stated above:

IBM United Kingdom Limited
Registered in England and Wales with number 741598
Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU

RE: [DISCUSS] FLIP-XXX Apicurio-avro format

Posted by David Radley <da...@uk.ibm.com>.
Hi Arvid,
Thank you very much for looking at this and for Robert to pass this through to you. Some quick thoughts.

The current design is not because it came from an earlier version of the fix, Chesnay and I came up with this  design, is because we took the assumptions that:

  *   the existing code serialise and deserialise methods that the Kafka connector is using.
  *   We wanted to use the existing Avro format – so do the Avro part of the serialization / deserialization
  *   The reasons we need the record is because the schema information is in the headers by default for Apicurio.

Happy to step back on this.
I was thinking if we want to change these assumptions, then the impact of the change would be bigger as we would need to reimagine the Kafka or Avro relationship with this format. Your idea of a new format factory for the deserialization and serialization, also sounds good.

On your solutions :
Re 1- this is what we have.
Re 2 and 3 I will investigate your suggestions. I like the sound of them and at first glance I don’t see anything obvious that you have not considered.

  *   If we can have a format specific factory that passes down the record for the payload for Apicurio. I just need to check that the metadata processing still works as expected.
  *   I assume that the default in the Factory will be to do what it currently does then for Apicurio we pass down the record. In this way it should not effect Confluent Avro.
  *   I assume that the connector knows about the format when it needs to do the serialization / deserialization – I need to check the code on this.
  *   I think the fix would basically be the same apart from the bespoke serialization – it would just pass down the record. And the format factory instead of class loading the new serialization / deserialization.
  *   I have not hardwired it into the Kafka connector. If we have the apicurio hard wired into the Kafka connector jar then those classes will be there when using Confluent Avro. I suggest we keep the format as a separate format jar, and SQL jar like the other formats.

On the if you want to be fancy – a variant of this was my first proposed fix 😊 .
   Kind regards, David.

From: Arvid Heise <ar...@apache.org>
Date: Tuesday, 24 September 2024 at 21:11
To: Robert Metzger <rm...@apache.org>
Cc: dev@flink.apache.org <de...@flink.apache.org>
Subject: [EXTERNAL] Re: [DISCUSS] FLIP-XXX Apicurio-avro format
Hi Robert, thanks for looping me in.

I have looked at the branch and the FLIP. Apicurio looks like a promising
alternative to Confluent SR and I'm certain that it's a good addition to
Flink.

However, at the current form it looks heavily overengineered. I'm
suspecting that comes from the attempt to do some generic implementation in
flink-core and later moving it into the kafka connector. I'm concerned for
two reasons: there is lots of code involved for a relatively minor change
that needs to be maintained and it looks like performance is going to
suffer from using SPI on the hotpath. I also don't fully understand why we
need to decide per record on which deserializer is the most suitable; I'd
expect one topic to either use Apicurio or not.

My proposal is to take a step back and ask what is really needed and
desired. I think from a high level, we need
1. a way for the user to specify that it's Apicurio and provide all the
config options
2. we need to modify the KafkaDynamicSource/Sink to use the respective
Apicurio (de)serializer
3. we need to implement the (de)serializer

Let's look at solutions
Re 1: This is straight-forward. I haven't seen this explicitly in the FLIP
and it might be worthwhile to call it out. It probably looks like this

CREATE TABLE user_created (
 ...
) WITH (

 'connector' = 'kafka',
 'topic' = 'user',
 'properties.bootstrap.servers' = '<broker>:9092',

 'key.format' = 'avro-apicurio',
 'key.avro-apicurio.url' = 'http://<apicurio>:8082<http://%3capicurio%3e:8082>',
 ...

 'value.format' = 'avro-apicurio',
 'value.avro-apicurio.url' = 'http://<apicurio>:8082<http://%3capicurio%3e:8082>',
 ...
)

Re 2: This is the hardest part as the KafkaDynamicSource/Sink basically
uses (De)serializationSchema which just knows bytes and can't use
ConsumerRecord. I see how you came to some conclusions.
However, I think there is still some easier way to achieve what you want:
We need to use the generified KafkaRecord(De)serializationSchema instead.
Let's check what needs to be done for KafkaDynamicSource (sink should be
similar):
* Replace all DeserializationSchema with KafkaRecordDeserializationSchema
in KafkaDynamicSource and DynamicKafkaDeserializationSchema
* Introduce a KafkaRecordDeserializationFormatFactory with an
implementation that delegates to existing factory and
uses KafkaRecordDeserializationSchema#valueOnly under the hood.
* Look in KafkaDynamicTableFactory for the old factory and wrap it into the
new factory. Optionally also look for the new factory (see below).

Re 3:
Implement Apicurio using KafkaRecordDeserializationSchema. This would be
directly usable in DataStream API.
For table API, you can implement the new
KafkaRecordDeserializationFormatFactory and dynamically find it. However,
as of now it seems like everything is in the same jar already. If it's
never really separated, I'm also fine to hardwire the Apicurio
implementations inside the KafkaDynamicTableFactory (so simply add it
statically to the SPI formats).

So as you can see, there is no need to introduce any kind of new SPI lookup
or did I miss something obvious?



Btw if we want to be fancy, an alternative/extended solution would be to
generify (De)serializationSchema in flink-core. A
RecordDeserializationSchema<R, T> would have a

void deserialize(R record, Collector<T> out) throws IOException;

and then you could plug in different kinds of records.
The existing DeserializationSchema would become

public interface DeserializationSchema<T> extends
RecordDeserializationSchema<byte[], T>

The KafkaRecordDeserializationSchema

public interface KafkaRecordDeserializationSchema<T> extends
RecordDeserializationSchema<ConsumerRecord<byte[], byte[]>, T>

However, that's a FLIP on its own, especially to maintain backwards
compatibility.

Best.

Arvid

PS: don't hesitate to reach out on slack for quicker feedback. we should
still primarily use the ML to keep everyone in the loop but it's usually
much faster to ping me there if you post an answer here.

On Tue, Sep 24, 2024 at 7:50 PM Robert Metzger <rm...@apache.org> wrote:

> @Arvid: Could you take a look at this.
>
> Thanks a lot.
>
> On Wed, Aug 14, 2024 at 6:19 PM David Radley <da...@uk.ibm.com>
> wrote:
>
>> Hi Danny,
>> Thank you for your feedback. I have brought a fork up to date with one
>> commit so you can
>> see the code; I need to test more scenarios and add more unit tests, but
>> it shows the basic idea of the design, Chesnay and I agreed.
>>
>>  Kind regards, David.
>>
>>
>> From: Danny Cranmer <da...@apache.org>
>> Date: Thursday, 25 July 2024 at 16:46
>> To: dev@flink.apache.org <de...@flink.apache.org>
>> Cc: David Radley <da...@uk.ibm.com>
>> Subject: [EXTERNAL] Re: [DISCUSS] FLIP-XXX Apicurio-avro format
>> Hey, Thanks for the updated FLIP. > 1. 20 or beyond Given this is
>> intended to go in the Kafka connector we can target the next Kafka
>> connector version (v3. 3. 0) for Flink 1. 18 and 1. 19 (or 1. 19 and 1. 20,
>> if 1. 20 is released first). > RecordBasedSerialization: : public
>>
>> Hey,
>>
>> Thanks for the updated FLIP.
>>
>> > 1.20 or beyond
>>
>> Given this is intended to go in the Kafka connector we can target the
>> next Kafka connector version (v3.3.0) for Flink 1.18 and 1.19 (or 1.19 and
>> 1.20, if 1.20 is released first).
>>
>>
>> >> Agreed
>>
>>  > RecordBasedSerialization::public boolean canProcess(byte[]
>> customSerialisation);
>>
>> Not sure what this is for, since we are passing a byte array and asking
>> if it can be serialized? Can you elaborate how the Kafka connector
>> interacts with these interfaces?
>> >> we are asking it can be processed. The canProcess has a default and
>> also can be overridden. The idea is that if we are in another format and
>> the apicurio jar is there then we might pick up the apicurio jar in the
>> discovery. So we check that what we have discovered really can process the
>> message. During trsting I have the key as raw and the value as apicurio,
>> and saw the issue that necessitates this method.
>>
>>
>> > byte[] customSerialisation
>>
>> There are a lot of byte arrays being passed around, I assume we are not
>> doing heavy conversions multiple times? Is there scope to introduce a new
>> intermediate type to prevent duplicate processing cycles?
>> Or even better, do you have a fork with these changes you can share?
>> >> https://github.com/davidradl/flink-connector-kafka/tree/FLINK-35311-3
>>
>> > Format 'avro-apicurio'
>>
>> Since this is Kafka only I would be tempted to include Kafka in this tag,
>> just in case we decide to add a Kafka agnostic format in the future.
>> Additionally this makes it clearer to the user it is Kafka only.
>> >> I was keeping this in line with the avro-confluent naming which is
>> also Kafka only. I do not have strong views on this – so I can change if we
>> think it is better.
>>
>> Thanks,
>> Danny
>>
>>
>>
>> On Thu, Jul 18, 2024 at 6:32 PM Martijn Visser <martijnvisser@apache.org
>> <ma...@apache.org>> wrote:
>> Hi David,
>>
>> The FLIP is updated.
>>
>> Cheers, Martijn
>>
>>
>> On Tue, Jul 16, 2024 at 6:33 PM David Radley <david_radley@uk.ibm.com
>> <ma...@uk.ibm.com>>
>> wrote:
>>
>> > Hello all,
>> >
>> >
>> >
>> > I have prototyped the new design and confirmed it works. I have
>> documented
>> > the design in google doc
>> >
>> https://docs.google.com/document/d/1J1E-cE-X2H3-kw4rNjLn71OGPQk_Yl1iGX4-eCHWLgE/edit
>> >
>> >
>> >
>> > By copy Martijn: please could you move this content over to replace the
>> > content of
>> >
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-454%3A+New+Apicurio+Avro+format
>> >
>> >
>> >
>> > As the design is significantly different, there will be a new period of
>> > discussion before going to vote.
>> >
>> >
>> >
>> > Fyi I will be on vacation after this week until the 13th of August.
>> >
>> >
>> >
>> > Kind regards, David.
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> > *From: *David Radley <david_radley@uk.ibm.com<mailto:
>> david_radley@uk.ibm.com>>
>> > *Date: *Tuesday, 9 July 2024 at 11:17
>> > *To: *dev@flink.apache.org<ma...@flink.apache.org> <
>> dev@flink.apache.org<ma...@flink.apache.org>>
>> > *Subject: *[EXTERNAL] RE: [DISCUSS] FLIP-XXX Apicurio-avro format
>> >
>> > Hi Kevin,
>> > I have agreed a design with Chesnay and Danny. I am implementing a
>> > prototype, to prove it works,  then will update the Flip text with the
>> new
>> > design. Initial testing is showing it working.
>> >
>> > Here is a quick history so you can understand our current thinking.
>> >
>> >   1.  Initially we passed maps for header information from the kafka
>> > connector to Flink for deserialization. Similar for serialize. This was
>> not
>> > great, because maps are not ideal and it was a big change as it needed
>> core
>> > Flink interface changes
>> >   2.  We then moved the Avro Apicurio format to the Kafka connector and
>> > looked to discover a new record based de/serialization interface. So we
>> > could pass down the record (containing the headers) rather than the
>> > payload. This did not work, because there is a dependence on the Avro
>> > connector that is not  aware of the new interface.
>> >   3.  We considered using Thread local storage to pass the headers, we
>> did
>> > not like this as there was a risk of memory leaks if we did not manage
>> the
>> > thread well, also the contract is hidden.
>> >   4.  We then came up with the current design that augments the
>> > deserialization in the Kafka connector in a new discovered record based
>> > deserialization, it then takes the headers out in the schema coder,
>> leaving
>> > the message as it was. Similar for serialization.
>> >
>> >
>> > One piece I need to work out the details of, is how to work when there
>> are
>> > 2 implementations that can be discovered, probably using an augmented
>> > format name as a factory identifier,
>> >
>> > I hope to put up a new design in the Flip by the end of next week, for
>> > wider review,
>> >     Kind regards, David.
>> >
>> >
>> > From: Kevin Lam <ke...@shopify.com.INVALID>
>> > Date: Monday, 8 July 2024 at 21:16
>> > To: dev@flink.apache.org<ma...@flink.apache.org> <
>> dev@flink.apache.org<ma...@flink.apache.org>>
>> > Subject: [EXTERNAL] Re: [DISCUSS] FLIP-XXX Apicurio-avro format
>> > Hi David,
>> >
>> > Any updates on the Kafka Message Header support? I am also interested in
>> > supporting headers with the Flink SQL Formats:
>> > https://lists.apache.org/thread/spl88o63sjm2dv4l5no0ym632d2yt2o6
>> >
>> > On Fri, Jun 14, 2024 at 6:10 AM David Radley <david_radley@uk.ibm.com
>> <ma...@uk.ibm.com>>
>> > wrote:
>> >
>> > > Hi everyone,
>> > > I have talked with Chesnay and Danny offline. Danny and I were not
>> very
>> > > happy with the passing Maps around, and were looking for a neater
>> design.
>> > > Chesnay suggested that we could move the new format to the Kafka
>> > connector,
>> > > then pass the Kafka record down to the deserialize logic so it can
>> make
>> > use
>> > > of the headers during deserialization and serialisation.
>> > >
>> > > I think this is a neat idea. This would mean:
>> > > - the Kafka connector code would need to be updated to pass down the
>> > Kafka
>> > > record
>> > > - there would be the Avro Apicurio format and SQL in the kafka
>> > repository.
>> > > We feel it is unlikely to want to use the Apicurio registry with
>> files,
>> > as
>> > > the Avro format could be used.
>> > >
>> > > Unfortunately I have found that this as not so straight forward to
>> > > implement as the Avro Apicurio format uses the Avro format, which is
>> tied
>> > > to the DeserializationSchema. We were hoping to have a new decoding
>> > > implementation that would pass down the Kafka record rather than the
>> > > payload. This does not appear possible without a Avro format change.
>> > >
>> > >
>> > > Inspired by this idea, I notice that
>> > > KafkaValueOnlyRecordDeserializerWrapper<T> extends
>> > > KafkaValueOnlyDeserializerWrapper
>> > >
>> > > Does
>> > >
>> > > deserializer.deserialize(record.topic(),record.value())
>> > >
>> > >
>> > >
>> > > I am investigating If I can add a factory/reflection to provide an
>> > > alternative
>> > > Implementation that will pass the record based (the kafka record is
>> not
>> > > serializable so I will pick what we need and deserialize) as a byte
>> > array.
>> > >
>> > > I would need to do this 4 times (value ,key for deserialisation and
>> > > serialisation. To do this I would need to convert the record into a
>> byte
>> > > array, so it fits into the existing interface
>> (DeserializationSchema).  I
>> > > think this could be a way through, to avoid using maps and avoid
>> changing
>> > > the existing Avro format and avoid change any core Flink interfaces.
>> > >
>> > > I am going to prototype this idea. WDYT?
>> > >
>> > > My thanks go to Chesnay and Danny for their support and insight around
>> > > this Flip,
>> > >    Kind regards, David.
>> > >
>> > >
>> > >
>> > >
>> > >
>> > >
>> > > From: David Radley <david_radley@uk.ibm.com<mailto:
>> david_radley@uk.ibm.com>>
>> > > Date: Wednesday, 29 May 2024 at 11:39
>> > > To: dev@flink.apache.org<ma...@flink.apache.org> <
>> dev@flink.apache.org<ma...@flink.apache.org>>
>> > > Subject: [EXTERNAL] RE: [DISCUSS] FLIP-XXX Apicurio-avro format
>> > > Hi Danny,
>> > > Thank you for your feedback on this.
>> > >
>> > > I agree that using maps has pros and cons. The maps are flexible, but
>> do
>> > > require the sender and receiver to know what is in the map.
>> > >
>> > > When you say “That sounds like it would fit in better, I assume we
>> cannot
>> > > just take that approach?” The motivation behind this Flip is to
>> support
>> > the
>> > > headers which is the usual way that Apicurio runs. We will support the
>> > > “schema id in the payload” as well.
>> > >
>> > > I agree with you when you say “ I am not 100% happy with the solution
>> > but I
>> > > cannot offer a better option.” – this is a pragmatic way we have
>> found to
>> > > solve this issue. I am open to any suggestions to improve this as
>> well.
>> > >
>> > > If we are going with the maps design (which is the best we have at the
>> > > moment) ; it would be good to have the Flink core changes in base
>> Flink
>> > > version 2.0 as this would mean we do not need to use reflection in a
>> > Flink
>> > > Kafka version 2 connector to work out if the runtime Flink has the new
>> > > methods.
>> > >
>> > > At this stage we only have one committer (yourself) backing this. Do
>> you
>> > > know of other 2 committers who would support this Flip?
>> > >
>> > >      Kind regards, David.
>> > >
>> > >
>> > >
>> > > From: Danny Cranmer <dannycranmer@apache.org<mailto:
>> dannycranmer@apache.org>>
>> > > Date: Friday, 24 May 2024 at 19:32
>> > > To: dev@flink.apache.org<ma...@flink.apache.org> <
>> dev@flink.apache.org<ma...@flink.apache.org>>
>> > > Subject: [EXTERNAL] Re: [DISCUSS] FLIP-XXX Apicurio-avro format
>> > > Hello,
>> > >
>> > > > I am curious what you mean by abused.
>> > >
>> > > I just meant we will end up adding more and more fields to this map
>> over
>> > > time, and it may be hard to undo.
>> > >
>> > > > For Apicurio it can be sent at the start of the payload like
>> Confluent
>> > > Avro does. Confluent Avro have a magic byte followed by 4 bytes of
>> schema
>> > > id, at the start of the payload. Apicurio clients and SerDe libraries
>> can
>> > > be configured to not put the schema id in the headers in which case
>> there
>> > > is a magic byte followed by an 8 byte schema at the start of the
>> payload.
>> > > In the deserialization case, we would not need to look at the headers
>> –
>> > > though the encoding is also in the headers.
>> > >
>> > > That sounds like it would fit in better, I assume we cannot just take
>> > that
>> > > approach?
>> > >
>> > > Thanks for the discussion. I am not 100% happy with the solution but I
>> > > cannot offer a better option. I would be interested to hear if others
>> > have
>> > > any suggestions. Playing devil's advocate against myself, we pass maps
>> > > around to configure connectors so it is not too far away from that.
>> > >
>> > > Thanks,
>> > > Danny
>> > >
>> > >
>> > > On Fri, May 24, 2024 at 2:23 PM David Radley <david_radley@uk.ibm.com
>> <ma...@uk.ibm.com>>
>> > > wrote:
>> > >
>> > > > Hi Danny,
>> > > > No worries, thanks for replying. I have working prototype code that
>> is
>> > > > being reviewed. It needs some cleaning up and more complete testing
>> > > before
>> > > > it is ready, but will give you the general idea [1][2] to help to
>> > assess
>> > > > this approach.
>> > > >
>> > > >
>> > > > I am curious what you mean by abused. I guess the approaches are
>> > between
>> > > > generic map, mechanism vs a more particular more granular things
>> being
>> > > > passed that might be used by another connector.
>> > > >
>> > > > Your first question:
>> > > > “how would this work if the schema ID is not in the Kafka headers,
>> as
>> > > > hinted to in the FLIP "usually the global ID in a Kafka header"?
>> > > >
>> > > > For Apicurio it can be sent at the start of the payload like
>> Confluent
>> > > > Avro does. Confluent Avro have a magic byte followed by 4 bytes of
>> > schema
>> > > > id, at the start of the payload. Apicurio clients and SerDe
>> libraries
>> > can
>> > > > be configured to not put the schema id in the headers in which case
>> > there
>> > > > is a magic byte followed by an 8 byte schema at the start of the
>> > payload.
>> > > > In the deserialization case, we would not need to look at the
>> headers –
>> > > > though the encoding is also in the headers.
>> > > >
>> > > > Your second question:
>> > > > “I am wondering if there are any other instances where the source
>> would
>> > > be
>> > > > aware of the schema ID and pass it through in this way?
>> > > > ”
>> > > > The examples I can think of are:
>> > > > - Avro can send the complete schema in a header, this is not
>> > recommended
>> > > > but in theory fits the need for a message payload to require
>> something
>> > > else
>> > > > to get the structure.
>> > > > - I see [2] that Apicurio Protobuf uses headers.
>> > > > - it might be that other message queuing projects like Rabbit MQ
>> would
>> > > > need this to be able to support Apicurio Avro & protobuf.
>> > > >
>> > > > Kind regards, David,
>> > > >
>> > > >
>> > > >
>> > > >
>> > > > [1] https://github.com/apache/flink/pull/24715
>> > > > [2] https://github.com/apache/flink-connector-kafka/pull/99
>> > > > [3]
>> > > >
>> > >
>> >
>> https://www.apicur.io/registry/docs/apicurio-registry/2.5.x/getting-started/assembly-configuring-kafka-client-serdes.html#registry-serdes-types-json_registry
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > > From: Danny Cranmer <dannycranmer@apache.org<mailto:
>> dannycranmer@apache.org>>
>> > > > Date: Friday, 24 May 2024 at 12:22
>> > > > To: dev@flink.apache.org<ma...@flink.apache.org> <
>> dev@flink.apache.org<ma...@flink.apache.org>>
>> > > > Subject: [EXTERNAL] Re: [DISCUSS] FLIP-XXX Apicurio-avro format
>> > > > Hello,
>> > > >
>> > > > Apologies, I am on vacation and have limited access to email.
>> > > >
>> > > > I can see the logic here and why you ended up where you did. I can
>> also
>> > > see
>> > > > there are other useful metadata fields that we might want to pass
>> > > through,
>> > > > which might result in this Map being abused (Kafka Topic, Kinesis
>> > Shard,
>> > > > etc).
>> > > >
>> > > > I have a follow up question, how would this work if the schema ID is
>> > not
>> > > in
>> > > > the Kafka headers, as hinted to in the FLIP "usually the global ID
>> in a
>> > > > Kafka header"? I am wondering if there are any other instances where
>> > the
>> > > > source would be aware of the schema ID and pass it through in this
>> way?
>> > > >
>> > > > Thanks,
>> > > > Danny
>> > > >
>> > > >
>> > > >
>> > > > On Wed, May 22, 2024 at 3:43 PM David Radley <
>> david_radley@uk.ibm.com<ma...@uk.ibm.com>>
>> > > > wrote:
>> > > >
>> > > > > Hi Danny,
>> > > > > Did you have a chance you have a look at my responses to your
>> > > feedback? I
>> > > > > am hoping to keep the momentum going on this one,   kind regards,
>> > > David.
>> > > > >
>> > > > >
>> > > > > From: David Radley <david_radley@uk.ibm.com<mailto:
>> david_radley@uk.ibm.com>>
>> > > > > Date: Tuesday, 14 May 2024 at 17:21
>> > > > > To: dev@flink.apache.org<ma...@flink.apache.org> <
>> dev@flink.apache.org<ma...@flink.apache.org>>
>> > > > > Subject: [EXTERNAL] [DISCUSS] FLIP-XXX Apicurio-avro format
>> > > > > Hi Danny,
>> > > > >
>> > > > > Thank you very much for the feedback and your support. I have
>> copied
>> > > your
>> > > > > feedback from the VOTE thread to this discussion thread, so we can
>> > > > continue
>> > > > > our discussions off the VOTE thread.
>> > > > >
>> > > > >
>> > > > >
>> > > > > Your feedback:
>> > > > >
>> > > > > Thanks for Driving this David. I am +1 for adding support for the
>> new
>> > > > >
>> > > > > format, however have some questions/suggestions on the details.
>> > > > >
>> > > > >
>> > > > >
>> > > > > 1. Passing around Map<String, Object> additionalInputProperties
>> > feels a
>> > > > bit
>> > > > >
>> > > > > dirty. It looks like this is mainly for the Kafka connector. This
>> > > > connector
>> > > > >
>> > > > > already has a de/serialization schema extension to access record
>> > > > >
>> > > > > headers, KafkaRecordDeserializationSchema [1], can we use this
>> > instead?
>> > > > >
>> > > > > 2. Can you elaborate why we need to change the SchemaCoder
>> interface?
>> > > > Again
>> > > > >
>> > > > > I am not a fan of adding these Map parameters
>> > > > >
>> > > > > 3. I assume this integration will go into the core Flink repo
>> under
>> > > > >
>> > > > > flink-formats [2], and not be a separate repository like the
>> > > connectors?
>> > > > >
>> > > > >
>> > > > >
>> > > > > My response:
>> > > > >
>> > > > > Addressing 1. and 2.
>> > > > >
>> > > > > I agree that sending maps around is a bit dirty. If we can see a
>> > better
>> > > > > way that would be great. I was looking for a way to pass this
>> kafka
>> > > > header
>> > > > > information in a non-Kafka way - the most obvious way I could
>> think
>> > was
>> > > > as
>> > > > > a map. Here are the main considerations I saw, if I have missed
>> > > anything
>> > > > or
>> > > > > could improve something I would be grateful for any further
>> feedback.
>> > > > >
>> > > > >
>> > > > >
>> > > > >   *   I see KafkaRecordDeserializationSchema is a Kafka interface
>> > that
>> > > > > works at the Kafka record level (so includes the headers). We
>> need a
>> > > > > mechanism to send over the headers from the Kafka record to Flink
>> > > > >   *   Flink core is not aware of Kafka headers, and I did not
>> want to
>> > > add
>> > > > > a Kafka dependancy to core flink.
>> > > > >   *   The formats are stateless so it did not appear to be in
>> fitting
>> > > > with
>> > > > > the Flink architecture to pass through header information to
>> stash in
>> > > > state
>> > > > > in the format waiting for the deserialise to be subsequently
>> called
>> > to
>> > > > pick
>> > > > > up the header information.
>> > > > >   *   We could have used Thread local storage to stash the header
>> > > > content,
>> > > > > but this would be extra state to manage; and this would seem like
>> an
>> > > > > obtrusive change.
>> > > > >   *   The SchemaCoder deserialise is where Confluent Avro gets the
>> > > schema
>> > > > > id from the payload, so it can lookup the schema. In line with
>> this
>> > > > > approach it made sense to extend the deserialise so it had the
>> header
>> > > > > contents so the Apicurio Avro format could lookup the schema.
>> > > > >   *   I did not want to have Apicurio specific logic in the Kafka
>> > > > > connector, if we did we could pull out the appropriate headers and
>> > only
>> > > > > send over the schema ids.
>> > > > >   *   For deserialise, the schema id we are interested in is the
>> one
>> > in
>> > > > > the Kafka headers on the message and is for the writer schema (an
>> > Avro
>> > > > > format concept) currently used by the confluent-avro format in
>> > > > deserialize.
>> > > > >   *   For serialise the schema ids need to be obtained from
>> apicurio
>> > > then
>> > > > > passed through to Kafka.
>> > > > >   *   For serialise there is existing logic around handling the
>> > > metadata
>> > > > > which includes passing the headers. But the presence of the
>> metadata
>> > > > would
>> > > > > imply we have a metadata column. Maybe a change to the metadata
>> > > mechanism
>> > > > > may have allowed to use to pass the headers, but not create a
>> > metadata
>> > > > > column; instead I pass through the additional headers in a map to
>> be
>> > > > > appended.
>> > > > >
>> > > > >
>> > > > >
>> > > > > 3.
>> > > > >
>> > > > > Yes this integration will go into the core Flink repo under
>> > > > >
>> > > > > flink-formats and sit next to the confluent-avro format. The Avro
>> > > format
>> > > > > has the concept of a Registry and drives the confluent-avro
>> format.
>> > The
>> > > > > Apicurio Avro format will use the same approach.
>> > > > >
>> > > > > Unless otherwise stated above:
>> > > > >
>> > > > > IBM United Kingdom Limited
>> > > > > Registered in England and Wales with number 741598
>> > > > > Registered office: PO Box 41, North Harbour, Portsmouth, Hants.
>> PO6
>> > 3AU
>> > > > >
>> > > > > Unless otherwise stated above:
>> > > > >
>> > > > > IBM United Kingdom Limited
>> > > > > Registered in England and Wales with number 741598
>> > > > > Registered office: PO Box 41, North Harbour, Portsmouth, Hants.
>> PO6
>> > 3AU
>> > > > >
>> > > >
>> > > > Unless otherwise stated above:
>> > > >
>> > > > IBM United Kingdom Limited
>> > > > Registered in England and Wales with number 741598
>> > > > Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6
>> 3AU
>> > > >
>> > >
>> > > Unless otherwise stated above:
>> > >
>> > > IBM United Kingdom Limited
>> > > Registered in England and Wales with number 741598
>> > > Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6
>> 3AU
>> > >
>> > > Unless otherwise stated above:
>> > >
>> > > IBM United Kingdom Limited
>> > > Registered in England and Wales with number 741598
>> > > Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6
>> 3AU
>> > >
>> >
>> > Unless otherwise stated above:
>> >
>> > IBM United Kingdom Limited
>> > Registered in England and Wales with number 741598
>> > Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU
>> > Unless otherwise stated above:
>> >
>> > IBM United Kingdom Limited
>> > Registered in England and Wales with number 741598
>> > Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU
>> >
>>
>> Unless otherwise stated above:
>>
>> IBM United Kingdom Limited
>> Registered in England and Wales with number 741598
>> Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU
>>
>

Unless otherwise stated above:

IBM United Kingdom Limited
Registered in England and Wales with number 741598
Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU

Re: [DISCUSS] FLIP-XXX Apicurio-avro format

Posted by Arvid Heise <ar...@apache.org>.
Hi Robert, thanks for looping me in.

I have looked at the branch and the FLIP. Apicurio looks like a promising
alternative to Confluent SR and I'm certain that it's a good addition to
Flink.

However, at the current form it looks heavily overengineered. I'm
suspecting that comes from the attempt to do some generic implementation in
flink-core and later moving it into the kafka connector. I'm concerned for
two reasons: there is lots of code involved for a relatively minor change
that needs to be maintained and it looks like performance is going to
suffer from using SPI on the hotpath. I also don't fully understand why we
need to decide per record on which deserializer is the most suitable; I'd
expect one topic to either use Apicurio or not.

My proposal is to take a step back and ask what is really needed and
desired. I think from a high level, we need
1. a way for the user to specify that it's Apicurio and provide all the
config options
2. we need to modify the KafkaDynamicSource/Sink to use the respective
Apicurio (de)serializer
3. we need to implement the (de)serializer

Let's look at solutions
Re 1: This is straight-forward. I haven't seen this explicitly in the FLIP
and it might be worthwhile to call it out. It probably looks like this

CREATE TABLE user_created (
 ...
) WITH (

 'connector' = 'kafka',
 'topic' = 'user',
 'properties.bootstrap.servers' = '<broker>:9092',

 'key.format' = 'avro-apicurio',
 'key.avro-apicurio.url' = 'http://<apicurio>:8082',
 ...

 'value.format' = 'avro-apicurio',
 'value.avro-apicurio.url' = 'http://<apicurio>:8082',
 ...
)

Re 2: This is the hardest part as the KafkaDynamicSource/Sink basically
uses (De)serializationSchema which just knows bytes and can't use
ConsumerRecord. I see how you came to some conclusions.
However, I think there is still some easier way to achieve what you want:
We need to use the generified KafkaRecord(De)serializationSchema instead.
Let's check what needs to be done for KafkaDynamicSource (sink should be
similar):
* Replace all DeserializationSchema with KafkaRecordDeserializationSchema
in KafkaDynamicSource and DynamicKafkaDeserializationSchema
* Introduce a KafkaRecordDeserializationFormatFactory with an
implementation that delegates to existing factory and
uses KafkaRecordDeserializationSchema#valueOnly under the hood.
* Look in KafkaDynamicTableFactory for the old factory and wrap it into the
new factory. Optionally also look for the new factory (see below).

Re 3:
Implement Apicurio using KafkaRecordDeserializationSchema. This would be
directly usable in DataStream API.
For table API, you can implement the new
KafkaRecordDeserializationFormatFactory and dynamically find it. However,
as of now it seems like everything is in the same jar already. If it's
never really separated, I'm also fine to hardwire the Apicurio
implementations inside the KafkaDynamicTableFactory (so simply add it
statically to the SPI formats).

So as you can see, there is no need to introduce any kind of new SPI lookup
or did I miss something obvious?



Btw if we want to be fancy, an alternative/extended solution would be to
generify (De)serializationSchema in flink-core. A
RecordDeserializationSchema<R, T> would have a

void deserialize(R record, Collector<T> out) throws IOException;

and then you could plug in different kinds of records.
The existing DeserializationSchema would become

public interface DeserializationSchema<T> extends
RecordDeserializationSchema<byte[], T>

The KafkaRecordDeserializationSchema

public interface KafkaRecordDeserializationSchema<T> extends
RecordDeserializationSchema<ConsumerRecord<byte[], byte[]>, T>

However, that's a FLIP on its own, especially to maintain backwards
compatibility.

Best.

Arvid

PS: don't hesitate to reach out on slack for quicker feedback. we should
still primarily use the ML to keep everyone in the loop but it's usually
much faster to ping me there if you post an answer here.

On Tue, Sep 24, 2024 at 7:50 PM Robert Metzger <rm...@apache.org> wrote:

> @Arvid: Could you take a look at this.
>
> Thanks a lot.
>
> On Wed, Aug 14, 2024 at 6:19 PM David Radley <da...@uk.ibm.com>
> wrote:
>
>> Hi Danny,
>> Thank you for your feedback. I have brought a fork up to date with one
>> commit so you can
>> see the code; I need to test more scenarios and add more unit tests, but
>> it shows the basic idea of the design, Chesnay and I agreed.
>>
>>  Kind regards, David.
>>
>>
>> From: Danny Cranmer <da...@apache.org>
>> Date: Thursday, 25 July 2024 at 16:46
>> To: dev@flink.apache.org <de...@flink.apache.org>
>> Cc: David Radley <da...@uk.ibm.com>
>> Subject: [EXTERNAL] Re: [DISCUSS] FLIP-XXX Apicurio-avro format
>> Hey, Thanks for the updated FLIP. > 1. 20 or beyond Given this is
>> intended to go in the Kafka connector we can target the next Kafka
>> connector version (v3. 3. 0) for Flink 1. 18 and 1. 19 (or 1. 19 and 1. 20,
>> if 1. 20 is released first). > RecordBasedSerialization: : public
>>
>> Hey,
>>
>> Thanks for the updated FLIP.
>>
>> > 1.20 or beyond
>>
>> Given this is intended to go in the Kafka connector we can target the
>> next Kafka connector version (v3.3.0) for Flink 1.18 and 1.19 (or 1.19 and
>> 1.20, if 1.20 is released first).
>>
>>
>> >> Agreed
>>
>>  > RecordBasedSerialization::public boolean canProcess(byte[]
>> customSerialisation);
>>
>> Not sure what this is for, since we are passing a byte array and asking
>> if it can be serialized? Can you elaborate how the Kafka connector
>> interacts with these interfaces?
>> >> we are asking it can be processed. The canProcess has a default and
>> also can be overridden. The idea is that if we are in another format and
>> the apicurio jar is there then we might pick up the apicurio jar in the
>> discovery. So we check that what we have discovered really can process the
>> message. During trsting I have the key as raw and the value as apicurio,
>> and saw the issue that necessitates this method.
>>
>>
>> > byte[] customSerialisation
>>
>> There are a lot of byte arrays being passed around, I assume we are not
>> doing heavy conversions multiple times? Is there scope to introduce a new
>> intermediate type to prevent duplicate processing cycles?
>> Or even better, do you have a fork with these changes you can share?
>> >> https://github.com/davidradl/flink-connector-kafka/tree/FLINK-35311-3
>>
>> > Format 'avro-apicurio'
>>
>> Since this is Kafka only I would be tempted to include Kafka in this tag,
>> just in case we decide to add a Kafka agnostic format in the future.
>> Additionally this makes it clearer to the user it is Kafka only.
>> >> I was keeping this in line with the avro-confluent naming which is
>> also Kafka only. I do not have strong views on this – so I can change if we
>> think it is better.
>>
>> Thanks,
>> Danny
>>
>>
>>
>> On Thu, Jul 18, 2024 at 6:32 PM Martijn Visser <martijnvisser@apache.org
>> <ma...@apache.org>> wrote:
>> Hi David,
>>
>> The FLIP is updated.
>>
>> Cheers, Martijn
>>
>>
>> On Tue, Jul 16, 2024 at 6:33 PM David Radley <david_radley@uk.ibm.com
>> <ma...@uk.ibm.com>>
>> wrote:
>>
>> > Hello all,
>> >
>> >
>> >
>> > I have prototyped the new design and confirmed it works. I have
>> documented
>> > the design in google doc
>> >
>> https://docs.google.com/document/d/1J1E-cE-X2H3-kw4rNjLn71OGPQk_Yl1iGX4-eCHWLgE/edit
>> >
>> >
>> >
>> > By copy Martijn: please could you move this content over to replace the
>> > content of
>> >
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-454%3A+New+Apicurio+Avro+format
>> >
>> >
>> >
>> > As the design is significantly different, there will be a new period of
>> > discussion before going to vote.
>> >
>> >
>> >
>> > Fyi I will be on vacation after this week until the 13th of August.
>> >
>> >
>> >
>> > Kind regards, David.
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> > *From: *David Radley <david_radley@uk.ibm.com<mailto:
>> david_radley@uk.ibm.com>>
>> > *Date: *Tuesday, 9 July 2024 at 11:17
>> > *To: *dev@flink.apache.org<ma...@flink.apache.org> <
>> dev@flink.apache.org<ma...@flink.apache.org>>
>> > *Subject: *[EXTERNAL] RE: [DISCUSS] FLIP-XXX Apicurio-avro format
>> >
>> > Hi Kevin,
>> > I have agreed a design with Chesnay and Danny. I am implementing a
>> > prototype, to prove it works,  then will update the Flip text with the
>> new
>> > design. Initial testing is showing it working.
>> >
>> > Here is a quick history so you can understand our current thinking.
>> >
>> >   1.  Initially we passed maps for header information from the kafka
>> > connector to Flink for deserialization. Similar for serialize. This was
>> not
>> > great, because maps are not ideal and it was a big change as it needed
>> core
>> > Flink interface changes
>> >   2.  We then moved the Avro Apicurio format to the Kafka connector and
>> > looked to discover a new record based de/serialization interface. So we
>> > could pass down the record (containing the headers) rather than the
>> > payload. This did not work, because there is a dependence on the Avro
>> > connector that is not  aware of the new interface.
>> >   3.  We considered using Thread local storage to pass the headers, we
>> did
>> > not like this as there was a risk of memory leaks if we did not manage
>> the
>> > thread well, also the contract is hidden.
>> >   4.  We then came up with the current design that augments the
>> > deserialization in the Kafka connector in a new discovered record based
>> > deserialization, it then takes the headers out in the schema coder,
>> leaving
>> > the message as it was. Similar for serialization.
>> >
>> >
>> > One piece I need to work out the details of, is how to work when there
>> are
>> > 2 implementations that can be discovered, probably using an augmented
>> > format name as a factory identifier,
>> >
>> > I hope to put up a new design in the Flip by the end of next week, for
>> > wider review,
>> >     Kind regards, David.
>> >
>> >
>> > From: Kevin Lam <ke...@shopify.com.INVALID>
>> > Date: Monday, 8 July 2024 at 21:16
>> > To: dev@flink.apache.org<ma...@flink.apache.org> <
>> dev@flink.apache.org<ma...@flink.apache.org>>
>> > Subject: [EXTERNAL] Re: [DISCUSS] FLIP-XXX Apicurio-avro format
>> > Hi David,
>> >
>> > Any updates on the Kafka Message Header support? I am also interested in
>> > supporting headers with the Flink SQL Formats:
>> > https://lists.apache.org/thread/spl88o63sjm2dv4l5no0ym632d2yt2o6
>> >
>> > On Fri, Jun 14, 2024 at 6:10 AM David Radley <david_radley@uk.ibm.com
>> <ma...@uk.ibm.com>>
>> > wrote:
>> >
>> > > Hi everyone,
>> > > I have talked with Chesnay and Danny offline. Danny and I were not
>> very
>> > > happy with the passing Maps around, and were looking for a neater
>> design.
>> > > Chesnay suggested that we could move the new format to the Kafka
>> > connector,
>> > > then pass the Kafka record down to the deserialize logic so it can
>> make
>> > use
>> > > of the headers during deserialization and serialisation.
>> > >
>> > > I think this is a neat idea. This would mean:
>> > > - the Kafka connector code would need to be updated to pass down the
>> > Kafka
>> > > record
>> > > - there would be the Avro Apicurio format and SQL in the kafka
>> > repository.
>> > > We feel it is unlikely to want to use the Apicurio registry with
>> files,
>> > as
>> > > the Avro format could be used.
>> > >
>> > > Unfortunately I have found that this as not so straight forward to
>> > > implement as the Avro Apicurio format uses the Avro format, which is
>> tied
>> > > to the DeserializationSchema. We were hoping to have a new decoding
>> > > implementation that would pass down the Kafka record rather than the
>> > > payload. This does not appear possible without a Avro format change.
>> > >
>> > >
>> > > Inspired by this idea, I notice that
>> > > KafkaValueOnlyRecordDeserializerWrapper<T> extends
>> > > KafkaValueOnlyDeserializerWrapper
>> > >
>> > > Does
>> > >
>> > > deserializer.deserialize(record.topic(),record.value())
>> > >
>> > >
>> > >
>> > > I am investigating If I can add a factory/reflection to provide an
>> > > alternative
>> > > Implementation that will pass the record based (the kafka record is
>> not
>> > > serializable so I will pick what we need and deserialize) as a byte
>> > array.
>> > >
>> > > I would need to do this 4 times (value ,key for deserialisation and
>> > > serialisation. To do this I would need to convert the record into a
>> byte
>> > > array, so it fits into the existing interface
>> (DeserializationSchema).  I
>> > > think this could be a way through, to avoid using maps and avoid
>> changing
>> > > the existing Avro format and avoid change any core Flink interfaces.
>> > >
>> > > I am going to prototype this idea. WDYT?
>> > >
>> > > My thanks go to Chesnay and Danny for their support and insight around
>> > > this Flip,
>> > >    Kind regards, David.
>> > >
>> > >
>> > >
>> > >
>> > >
>> > >
>> > > From: David Radley <david_radley@uk.ibm.com<mailto:
>> david_radley@uk.ibm.com>>
>> > > Date: Wednesday, 29 May 2024 at 11:39
>> > > To: dev@flink.apache.org<ma...@flink.apache.org> <
>> dev@flink.apache.org<ma...@flink.apache.org>>
>> > > Subject: [EXTERNAL] RE: [DISCUSS] FLIP-XXX Apicurio-avro format
>> > > Hi Danny,
>> > > Thank you for your feedback on this.
>> > >
>> > > I agree that using maps has pros and cons. The maps are flexible, but
>> do
>> > > require the sender and receiver to know what is in the map.
>> > >
>> > > When you say “That sounds like it would fit in better, I assume we
>> cannot
>> > > just take that approach?” The motivation behind this Flip is to
>> support
>> > the
>> > > headers which is the usual way that Apicurio runs. We will support the
>> > > “schema id in the payload” as well.
>> > >
>> > > I agree with you when you say “ I am not 100% happy with the solution
>> > but I
>> > > cannot offer a better option.” – this is a pragmatic way we have
>> found to
>> > > solve this issue. I am open to any suggestions to improve this as
>> well.
>> > >
>> > > If we are going with the maps design (which is the best we have at the
>> > > moment) ; it would be good to have the Flink core changes in base
>> Flink
>> > > version 2.0 as this would mean we do not need to use reflection in a
>> > Flink
>> > > Kafka version 2 connector to work out if the runtime Flink has the new
>> > > methods.
>> > >
>> > > At this stage we only have one committer (yourself) backing this. Do
>> you
>> > > know of other 2 committers who would support this Flip?
>> > >
>> > >      Kind regards, David.
>> > >
>> > >
>> > >
>> > > From: Danny Cranmer <dannycranmer@apache.org<mailto:
>> dannycranmer@apache.org>>
>> > > Date: Friday, 24 May 2024 at 19:32
>> > > To: dev@flink.apache.org<ma...@flink.apache.org> <
>> dev@flink.apache.org<ma...@flink.apache.org>>
>> > > Subject: [EXTERNAL] Re: [DISCUSS] FLIP-XXX Apicurio-avro format
>> > > Hello,
>> > >
>> > > > I am curious what you mean by abused.
>> > >
>> > > I just meant we will end up adding more and more fields to this map
>> over
>> > > time, and it may be hard to undo.
>> > >
>> > > > For Apicurio it can be sent at the start of the payload like
>> Confluent
>> > > Avro does. Confluent Avro have a magic byte followed by 4 bytes of
>> schema
>> > > id, at the start of the payload. Apicurio clients and SerDe libraries
>> can
>> > > be configured to not put the schema id in the headers in which case
>> there
>> > > is a magic byte followed by an 8 byte schema at the start of the
>> payload.
>> > > In the deserialization case, we would not need to look at the headers
>> –
>> > > though the encoding is also in the headers.
>> > >
>> > > That sounds like it would fit in better, I assume we cannot just take
>> > that
>> > > approach?
>> > >
>> > > Thanks for the discussion. I am not 100% happy with the solution but I
>> > > cannot offer a better option. I would be interested to hear if others
>> > have
>> > > any suggestions. Playing devil's advocate against myself, we pass maps
>> > > around to configure connectors so it is not too far away from that.
>> > >
>> > > Thanks,
>> > > Danny
>> > >
>> > >
>> > > On Fri, May 24, 2024 at 2:23 PM David Radley <david_radley@uk.ibm.com
>> <ma...@uk.ibm.com>>
>> > > wrote:
>> > >
>> > > > Hi Danny,
>> > > > No worries, thanks for replying. I have working prototype code that
>> is
>> > > > being reviewed. It needs some cleaning up and more complete testing
>> > > before
>> > > > it is ready, but will give you the general idea [1][2] to help to
>> > assess
>> > > > this approach.
>> > > >
>> > > >
>> > > > I am curious what you mean by abused. I guess the approaches are
>> > between
>> > > > generic map, mechanism vs a more particular more granular things
>> being
>> > > > passed that might be used by another connector.
>> > > >
>> > > > Your first question:
>> > > > “how would this work if the schema ID is not in the Kafka headers,
>> as
>> > > > hinted to in the FLIP "usually the global ID in a Kafka header"?
>> > > >
>> > > > For Apicurio it can be sent at the start of the payload like
>> Confluent
>> > > > Avro does. Confluent Avro have a magic byte followed by 4 bytes of
>> > schema
>> > > > id, at the start of the payload. Apicurio clients and SerDe
>> libraries
>> > can
>> > > > be configured to not put the schema id in the headers in which case
>> > there
>> > > > is a magic byte followed by an 8 byte schema at the start of the
>> > payload.
>> > > > In the deserialization case, we would not need to look at the
>> headers –
>> > > > though the encoding is also in the headers.
>> > > >
>> > > > Your second question:
>> > > > “I am wondering if there are any other instances where the source
>> would
>> > > be
>> > > > aware of the schema ID and pass it through in this way?
>> > > > ”
>> > > > The examples I can think of are:
>> > > > - Avro can send the complete schema in a header, this is not
>> > recommended
>> > > > but in theory fits the need for a message payload to require
>> something
>> > > else
>> > > > to get the structure.
>> > > > - I see [2] that Apicurio Protobuf uses headers.
>> > > > - it might be that other message queuing projects like Rabbit MQ
>> would
>> > > > need this to be able to support Apicurio Avro & protobuf.
>> > > >
>> > > > Kind regards, David,
>> > > >
>> > > >
>> > > >
>> > > >
>> > > > [1] https://github.com/apache/flink/pull/24715
>> > > > [2] https://github.com/apache/flink-connector-kafka/pull/99
>> > > > [3]
>> > > >
>> > >
>> >
>> https://www.apicur.io/registry/docs/apicurio-registry/2.5.x/getting-started/assembly-configuring-kafka-client-serdes.html#registry-serdes-types-json_registry
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > > From: Danny Cranmer <dannycranmer@apache.org<mailto:
>> dannycranmer@apache.org>>
>> > > > Date: Friday, 24 May 2024 at 12:22
>> > > > To: dev@flink.apache.org<ma...@flink.apache.org> <
>> dev@flink.apache.org<ma...@flink.apache.org>>
>> > > > Subject: [EXTERNAL] Re: [DISCUSS] FLIP-XXX Apicurio-avro format
>> > > > Hello,
>> > > >
>> > > > Apologies, I am on vacation and have limited access to email.
>> > > >
>> > > > I can see the logic here and why you ended up where you did. I can
>> also
>> > > see
>> > > > there are other useful metadata fields that we might want to pass
>> > > through,
>> > > > which might result in this Map being abused (Kafka Topic, Kinesis
>> > Shard,
>> > > > etc).
>> > > >
>> > > > I have a follow up question, how would this work if the schema ID is
>> > not
>> > > in
>> > > > the Kafka headers, as hinted to in the FLIP "usually the global ID
>> in a
>> > > > Kafka header"? I am wondering if there are any other instances where
>> > the
>> > > > source would be aware of the schema ID and pass it through in this
>> way?
>> > > >
>> > > > Thanks,
>> > > > Danny
>> > > >
>> > > >
>> > > >
>> > > > On Wed, May 22, 2024 at 3:43 PM David Radley <
>> david_radley@uk.ibm.com<ma...@uk.ibm.com>>
>> > > > wrote:
>> > > >
>> > > > > Hi Danny,
>> > > > > Did you have a chance you have a look at my responses to your
>> > > feedback? I
>> > > > > am hoping to keep the momentum going on this one,   kind regards,
>> > > David.
>> > > > >
>> > > > >
>> > > > > From: David Radley <david_radley@uk.ibm.com<mailto:
>> david_radley@uk.ibm.com>>
>> > > > > Date: Tuesday, 14 May 2024 at 17:21
>> > > > > To: dev@flink.apache.org<ma...@flink.apache.org> <
>> dev@flink.apache.org<ma...@flink.apache.org>>
>> > > > > Subject: [EXTERNAL] [DISCUSS] FLIP-XXX Apicurio-avro format
>> > > > > Hi Danny,
>> > > > >
>> > > > > Thank you very much for the feedback and your support. I have
>> copied
>> > > your
>> > > > > feedback from the VOTE thread to this discussion thread, so we can
>> > > > continue
>> > > > > our discussions off the VOTE thread.
>> > > > >
>> > > > >
>> > > > >
>> > > > > Your feedback:
>> > > > >
>> > > > > Thanks for Driving this David. I am +1 for adding support for the
>> new
>> > > > >
>> > > > > format, however have some questions/suggestions on the details.
>> > > > >
>> > > > >
>> > > > >
>> > > > > 1. Passing around Map<String, Object> additionalInputProperties
>> > feels a
>> > > > bit
>> > > > >
>> > > > > dirty. It looks like this is mainly for the Kafka connector. This
>> > > > connector
>> > > > >
>> > > > > already has a de/serialization schema extension to access record
>> > > > >
>> > > > > headers, KafkaRecordDeserializationSchema [1], can we use this
>> > instead?
>> > > > >
>> > > > > 2. Can you elaborate why we need to change the SchemaCoder
>> interface?
>> > > > Again
>> > > > >
>> > > > > I am not a fan of adding these Map parameters
>> > > > >
>> > > > > 3. I assume this integration will go into the core Flink repo
>> under
>> > > > >
>> > > > > flink-formats [2], and not be a separate repository like the
>> > > connectors?
>> > > > >
>> > > > >
>> > > > >
>> > > > > My response:
>> > > > >
>> > > > > Addressing 1. and 2.
>> > > > >
>> > > > > I agree that sending maps around is a bit dirty. If we can see a
>> > better
>> > > > > way that would be great. I was looking for a way to pass this
>> kafka
>> > > > header
>> > > > > information in a non-Kafka way - the most obvious way I could
>> think
>> > was
>> > > > as
>> > > > > a map. Here are the main considerations I saw, if I have missed
>> > > anything
>> > > > or
>> > > > > could improve something I would be grateful for any further
>> feedback.
>> > > > >
>> > > > >
>> > > > >
>> > > > >   *   I see KafkaRecordDeserializationSchema is a Kafka interface
>> > that
>> > > > > works at the Kafka record level (so includes the headers). We
>> need a
>> > > > > mechanism to send over the headers from the Kafka record to Flink
>> > > > >   *   Flink core is not aware of Kafka headers, and I did not
>> want to
>> > > add
>> > > > > a Kafka dependancy to core flink.
>> > > > >   *   The formats are stateless so it did not appear to be in
>> fitting
>> > > > with
>> > > > > the Flink architecture to pass through header information to
>> stash in
>> > > > state
>> > > > > in the format waiting for the deserialise to be subsequently
>> called
>> > to
>> > > > pick
>> > > > > up the header information.
>> > > > >   *   We could have used Thread local storage to stash the header
>> > > > content,
>> > > > > but this would be extra state to manage; and this would seem like
>> an
>> > > > > obtrusive change.
>> > > > >   *   The SchemaCoder deserialise is where Confluent Avro gets the
>> > > schema
>> > > > > id from the payload, so it can lookup the schema. In line with
>> this
>> > > > > approach it made sense to extend the deserialise so it had the
>> header
>> > > > > contents so the Apicurio Avro format could lookup the schema.
>> > > > >   *   I did not want to have Apicurio specific logic in the Kafka
>> > > > > connector, if we did we could pull out the appropriate headers and
>> > only
>> > > > > send over the schema ids.
>> > > > >   *   For deserialise, the schema id we are interested in is the
>> one
>> > in
>> > > > > the Kafka headers on the message and is for the writer schema (an
>> > Avro
>> > > > > format concept) currently used by the confluent-avro format in
>> > > > deserialize.
>> > > > >   *   For serialise the schema ids need to be obtained from
>> apicurio
>> > > then
>> > > > > passed through to Kafka.
>> > > > >   *   For serialise there is existing logic around handling the
>> > > metadata
>> > > > > which includes passing the headers. But the presence of the
>> metadata
>> > > > would
>> > > > > imply we have a metadata column. Maybe a change to the metadata
>> > > mechanism
>> > > > > may have allowed to use to pass the headers, but not create a
>> > metadata
>> > > > > column; instead I pass through the additional headers in a map to
>> be
>> > > > > appended.
>> > > > >
>> > > > >
>> > > > >
>> > > > > 3.
>> > > > >
>> > > > > Yes this integration will go into the core Flink repo under
>> > > > >
>> > > > > flink-formats and sit next to the confluent-avro format. The Avro
>> > > format
>> > > > > has the concept of a Registry and drives the confluent-avro
>> format.
>> > The
>> > > > > Apicurio Avro format will use the same approach.
>> > > > >
>> > > > > Unless otherwise stated above:
>> > > > >
>> > > > > IBM United Kingdom Limited
>> > > > > Registered in England and Wales with number 741598
>> > > > > Registered office: PO Box 41, North Harbour, Portsmouth, Hants.
>> PO6
>> > 3AU
>> > > > >
>> > > > > Unless otherwise stated above:
>> > > > >
>> > > > > IBM United Kingdom Limited
>> > > > > Registered in England and Wales with number 741598
>> > > > > Registered office: PO Box 41, North Harbour, Portsmouth, Hants.
>> PO6
>> > 3AU
>> > > > >
>> > > >
>> > > > Unless otherwise stated above:
>> > > >
>> > > > IBM United Kingdom Limited
>> > > > Registered in England and Wales with number 741598
>> > > > Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6
>> 3AU
>> > > >
>> > >
>> > > Unless otherwise stated above:
>> > >
>> > > IBM United Kingdom Limited
>> > > Registered in England and Wales with number 741598
>> > > Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6
>> 3AU
>> > >
>> > > Unless otherwise stated above:
>> > >
>> > > IBM United Kingdom Limited
>> > > Registered in England and Wales with number 741598
>> > > Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6
>> 3AU
>> > >
>> >
>> > Unless otherwise stated above:
>> >
>> > IBM United Kingdom Limited
>> > Registered in England and Wales with number 741598
>> > Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU
>> > Unless otherwise stated above:
>> >
>> > IBM United Kingdom Limited
>> > Registered in England and Wales with number 741598
>> > Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU
>> >
>>
>> Unless otherwise stated above:
>>
>> IBM United Kingdom Limited
>> Registered in England and Wales with number 741598
>> Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU
>>
>

Re: [DISCUSS] FLIP-XXX Apicurio-avro format

Posted by Robert Metzger <rm...@apache.org>.
@Arvid: Could you take a look at this.

Thanks a lot.

On Wed, Aug 14, 2024 at 6:19 PM David Radley <da...@uk.ibm.com>
wrote:

> Hi Danny,
> Thank you for your feedback. I have brought a fork up to date with one
> commit so you can
> see the code; I need to test more scenarios and add more unit tests, but
> it shows the basic idea of the design, Chesnay and I agreed.
>
>  Kind regards, David.
>
>
> From: Danny Cranmer <da...@apache.org>
> Date: Thursday, 25 July 2024 at 16:46
> To: dev@flink.apache.org <de...@flink.apache.org>
> Cc: David Radley <da...@uk.ibm.com>
> Subject: [EXTERNAL] Re: [DISCUSS] FLIP-XXX Apicurio-avro format
> Hey, Thanks for the updated FLIP. > 1. 20 or beyond Given this is intended
> to go in the Kafka connector we can target the next Kafka connector version
> (v3. 3. 0) for Flink 1. 18 and 1. 19 (or 1. 19 and 1. 20, if 1. 20 is
> released first). > RecordBasedSerialization: : public
>
> Hey,
>
> Thanks for the updated FLIP.
>
> > 1.20 or beyond
>
> Given this is intended to go in the Kafka connector we can target the next
> Kafka connector version (v3.3.0) for Flink 1.18 and 1.19 (or 1.19 and 1.20,
> if 1.20 is released first).
>
>
> >> Agreed
>
>  > RecordBasedSerialization::public boolean canProcess(byte[]
> customSerialisation);
>
> Not sure what this is for, since we are passing a byte array and asking if
> it can be serialized? Can you elaborate how the Kafka connector interacts
> with these interfaces?
> >> we are asking it can be processed. The canProcess has a default and
> also can be overridden. The idea is that if we are in another format and
> the apicurio jar is there then we might pick up the apicurio jar in the
> discovery. So we check that what we have discovered really can process the
> message. During trsting I have the key as raw and the value as apicurio,
> and saw the issue that necessitates this method.
>
>
> > byte[] customSerialisation
>
> There are a lot of byte arrays being passed around, I assume we are not
> doing heavy conversions multiple times? Is there scope to introduce a new
> intermediate type to prevent duplicate processing cycles?
> Or even better, do you have a fork with these changes you can share?
> >> https://github.com/davidradl/flink-connector-kafka/tree/FLINK-35311-3
>
> > Format 'avro-apicurio'
>
> Since this is Kafka only I would be tempted to include Kafka in this tag,
> just in case we decide to add a Kafka agnostic format in the future.
> Additionally this makes it clearer to the user it is Kafka only.
> >> I was keeping this in line with the avro-confluent naming which is also
> Kafka only. I do not have strong views on this – so I can change if we
> think it is better.
>
> Thanks,
> Danny
>
>
>
> On Thu, Jul 18, 2024 at 6:32 PM Martijn Visser <martijnvisser@apache.org
> <ma...@apache.org>> wrote:
> Hi David,
>
> The FLIP is updated.
>
> Cheers, Martijn
>
>
> On Tue, Jul 16, 2024 at 6:33 PM David Radley <david_radley@uk.ibm.com
> <ma...@uk.ibm.com>>
> wrote:
>
> > Hello all,
> >
> >
> >
> > I have prototyped the new design and confirmed it works. I have
> documented
> > the design in google doc
> >
> https://docs.google.com/document/d/1J1E-cE-X2H3-kw4rNjLn71OGPQk_Yl1iGX4-eCHWLgE/edit
> >
> >
> >
> > By copy Martijn: please could you move this content over to replace the
> > content of
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-454%3A+New+Apicurio+Avro+format
> >
> >
> >
> > As the design is significantly different, there will be a new period of
> > discussion before going to vote.
> >
> >
> >
> > Fyi I will be on vacation after this week until the 13th of August.
> >
> >
> >
> > Kind regards, David.
> >
> >
> >
> >
> >
> >
> >
> > *From: *David Radley <david_radley@uk.ibm.com<mailto:
> david_radley@uk.ibm.com>>
> > *Date: *Tuesday, 9 July 2024 at 11:17
> > *To: *dev@flink.apache.org<ma...@flink.apache.org> <
> dev@flink.apache.org<ma...@flink.apache.org>>
> > *Subject: *[EXTERNAL] RE: [DISCUSS] FLIP-XXX Apicurio-avro format
> >
> > Hi Kevin,
> > I have agreed a design with Chesnay and Danny. I am implementing a
> > prototype, to prove it works,  then will update the Flip text with the
> new
> > design. Initial testing is showing it working.
> >
> > Here is a quick history so you can understand our current thinking.
> >
> >   1.  Initially we passed maps for header information from the kafka
> > connector to Flink for deserialization. Similar for serialize. This was
> not
> > great, because maps are not ideal and it was a big change as it needed
> core
> > Flink interface changes
> >   2.  We then moved the Avro Apicurio format to the Kafka connector and
> > looked to discover a new record based de/serialization interface. So we
> > could pass down the record (containing the headers) rather than the
> > payload. This did not work, because there is a dependence on the Avro
> > connector that is not  aware of the new interface.
> >   3.  We considered using Thread local storage to pass the headers, we
> did
> > not like this as there was a risk of memory leaks if we did not manage
> the
> > thread well, also the contract is hidden.
> >   4.  We then came up with the current design that augments the
> > deserialization in the Kafka connector in a new discovered record based
> > deserialization, it then takes the headers out in the schema coder,
> leaving
> > the message as it was. Similar for serialization.
> >
> >
> > One piece I need to work out the details of, is how to work when there
> are
> > 2 implementations that can be discovered, probably using an augmented
> > format name as a factory identifier,
> >
> > I hope to put up a new design in the Flip by the end of next week, for
> > wider review,
> >     Kind regards, David.
> >
> >
> > From: Kevin Lam <ke...@shopify.com.INVALID>
> > Date: Monday, 8 July 2024 at 21:16
> > To: dev@flink.apache.org<ma...@flink.apache.org> <
> dev@flink.apache.org<ma...@flink.apache.org>>
> > Subject: [EXTERNAL] Re: [DISCUSS] FLIP-XXX Apicurio-avro format
> > Hi David,
> >
> > Any updates on the Kafka Message Header support? I am also interested in
> > supporting headers with the Flink SQL Formats:
> > https://lists.apache.org/thread/spl88o63sjm2dv4l5no0ym632d2yt2o6
> >
> > On Fri, Jun 14, 2024 at 6:10 AM David Radley <david_radley@uk.ibm.com
> <ma...@uk.ibm.com>>
> > wrote:
> >
> > > Hi everyone,
> > > I have talked with Chesnay and Danny offline. Danny and I were not very
> > > happy with the passing Maps around, and were looking for a neater
> design.
> > > Chesnay suggested that we could move the new format to the Kafka
> > connector,
> > > then pass the Kafka record down to the deserialize logic so it can make
> > use
> > > of the headers during deserialization and serialisation.
> > >
> > > I think this is a neat idea. This would mean:
> > > - the Kafka connector code would need to be updated to pass down the
> > Kafka
> > > record
> > > - there would be the Avro Apicurio format and SQL in the kafka
> > repository.
> > > We feel it is unlikely to want to use the Apicurio registry with files,
> > as
> > > the Avro format could be used.
> > >
> > > Unfortunately I have found that this as not so straight forward to
> > > implement as the Avro Apicurio format uses the Avro format, which is
> tied
> > > to the DeserializationSchema. We were hoping to have a new decoding
> > > implementation that would pass down the Kafka record rather than the
> > > payload. This does not appear possible without a Avro format change.
> > >
> > >
> > > Inspired by this idea, I notice that
> > > KafkaValueOnlyRecordDeserializerWrapper<T> extends
> > > KafkaValueOnlyDeserializerWrapper
> > >
> > > Does
> > >
> > > deserializer.deserialize(record.topic(),record.value())
> > >
> > >
> > >
> > > I am investigating If I can add a factory/reflection to provide an
> > > alternative
> > > Implementation that will pass the record based (the kafka record is not
> > > serializable so I will pick what we need and deserialize) as a byte
> > array.
> > >
> > > I would need to do this 4 times (value ,key for deserialisation and
> > > serialisation. To do this I would need to convert the record into a
> byte
> > > array, so it fits into the existing interface
> (DeserializationSchema).  I
> > > think this could be a way through, to avoid using maps and avoid
> changing
> > > the existing Avro format and avoid change any core Flink interfaces.
> > >
> > > I am going to prototype this idea. WDYT?
> > >
> > > My thanks go to Chesnay and Danny for their support and insight around
> > > this Flip,
> > >    Kind regards, David.
> > >
> > >
> > >
> > >
> > >
> > >
> > > From: David Radley <david_radley@uk.ibm.com<mailto:
> david_radley@uk.ibm.com>>
> > > Date: Wednesday, 29 May 2024 at 11:39
> > > To: dev@flink.apache.org<ma...@flink.apache.org> <
> dev@flink.apache.org<ma...@flink.apache.org>>
> > > Subject: [EXTERNAL] RE: [DISCUSS] FLIP-XXX Apicurio-avro format
> > > Hi Danny,
> > > Thank you for your feedback on this.
> > >
> > > I agree that using maps has pros and cons. The maps are flexible, but
> do
> > > require the sender and receiver to know what is in the map.
> > >
> > > When you say “That sounds like it would fit in better, I assume we
> cannot
> > > just take that approach?” The motivation behind this Flip is to support
> > the
> > > headers which is the usual way that Apicurio runs. We will support the
> > > “schema id in the payload” as well.
> > >
> > > I agree with you when you say “ I am not 100% happy with the solution
> > but I
> > > cannot offer a better option.” – this is a pragmatic way we have found
> to
> > > solve this issue. I am open to any suggestions to improve this as well.
> > >
> > > If we are going with the maps design (which is the best we have at the
> > > moment) ; it would be good to have the Flink core changes in base Flink
> > > version 2.0 as this would mean we do not need to use reflection in a
> > Flink
> > > Kafka version 2 connector to work out if the runtime Flink has the new
> > > methods.
> > >
> > > At this stage we only have one committer (yourself) backing this. Do
> you
> > > know of other 2 committers who would support this Flip?
> > >
> > >      Kind regards, David.
> > >
> > >
> > >
> > > From: Danny Cranmer <dannycranmer@apache.org<mailto:
> dannycranmer@apache.org>>
> > > Date: Friday, 24 May 2024 at 19:32
> > > To: dev@flink.apache.org<ma...@flink.apache.org> <
> dev@flink.apache.org<ma...@flink.apache.org>>
> > > Subject: [EXTERNAL] Re: [DISCUSS] FLIP-XXX Apicurio-avro format
> > > Hello,
> > >
> > > > I am curious what you mean by abused.
> > >
> > > I just meant we will end up adding more and more fields to this map
> over
> > > time, and it may be hard to undo.
> > >
> > > > For Apicurio it can be sent at the start of the payload like
> Confluent
> > > Avro does. Confluent Avro have a magic byte followed by 4 bytes of
> schema
> > > id, at the start of the payload. Apicurio clients and SerDe libraries
> can
> > > be configured to not put the schema id in the headers in which case
> there
> > > is a magic byte followed by an 8 byte schema at the start of the
> payload.
> > > In the deserialization case, we would not need to look at the headers –
> > > though the encoding is also in the headers.
> > >
> > > That sounds like it would fit in better, I assume we cannot just take
> > that
> > > approach?
> > >
> > > Thanks for the discussion. I am not 100% happy with the solution but I
> > > cannot offer a better option. I would be interested to hear if others
> > have
> > > any suggestions. Playing devil's advocate against myself, we pass maps
> > > around to configure connectors so it is not too far away from that.
> > >
> > > Thanks,
> > > Danny
> > >
> > >
> > > On Fri, May 24, 2024 at 2:23 PM David Radley <david_radley@uk.ibm.com
> <ma...@uk.ibm.com>>
> > > wrote:
> > >
> > > > Hi Danny,
> > > > No worries, thanks for replying. I have working prototype code that
> is
> > > > being reviewed. It needs some cleaning up and more complete testing
> > > before
> > > > it is ready, but will give you the general idea [1][2] to help to
> > assess
> > > > this approach.
> > > >
> > > >
> > > > I am curious what you mean by abused. I guess the approaches are
> > between
> > > > generic map, mechanism vs a more particular more granular things
> being
> > > > passed that might be used by another connector.
> > > >
> > > > Your first question:
> > > > “how would this work if the schema ID is not in the Kafka headers, as
> > > > hinted to in the FLIP "usually the global ID in a Kafka header"?
> > > >
> > > > For Apicurio it can be sent at the start of the payload like
> Confluent
> > > > Avro does. Confluent Avro have a magic byte followed by 4 bytes of
> > schema
> > > > id, at the start of the payload. Apicurio clients and SerDe libraries
> > can
> > > > be configured to not put the schema id in the headers in which case
> > there
> > > > is a magic byte followed by an 8 byte schema at the start of the
> > payload.
> > > > In the deserialization case, we would not need to look at the
> headers –
> > > > though the encoding is also in the headers.
> > > >
> > > > Your second question:
> > > > “I am wondering if there are any other instances where the source
> would
> > > be
> > > > aware of the schema ID and pass it through in this way?
> > > > ”
> > > > The examples I can think of are:
> > > > - Avro can send the complete schema in a header, this is not
> > recommended
> > > > but in theory fits the need for a message payload to require
> something
> > > else
> > > > to get the structure.
> > > > - I see [2] that Apicurio Protobuf uses headers.
> > > > - it might be that other message queuing projects like Rabbit MQ
> would
> > > > need this to be able to support Apicurio Avro & protobuf.
> > > >
> > > > Kind regards, David,
> > > >
> > > >
> > > >
> > > >
> > > > [1] https://github.com/apache/flink/pull/24715
> > > > [2] https://github.com/apache/flink-connector-kafka/pull/99
> > > > [3]
> > > >
> > >
> >
> https://www.apicur.io/registry/docs/apicurio-registry/2.5.x/getting-started/assembly-configuring-kafka-client-serdes.html#registry-serdes-types-json_registry
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > From: Danny Cranmer <dannycranmer@apache.org<mailto:
> dannycranmer@apache.org>>
> > > > Date: Friday, 24 May 2024 at 12:22
> > > > To: dev@flink.apache.org<ma...@flink.apache.org> <
> dev@flink.apache.org<ma...@flink.apache.org>>
> > > > Subject: [EXTERNAL] Re: [DISCUSS] FLIP-XXX Apicurio-avro format
> > > > Hello,
> > > >
> > > > Apologies, I am on vacation and have limited access to email.
> > > >
> > > > I can see the logic here and why you ended up where you did. I can
> also
> > > see
> > > > there are other useful metadata fields that we might want to pass
> > > through,
> > > > which might result in this Map being abused (Kafka Topic, Kinesis
> > Shard,
> > > > etc).
> > > >
> > > > I have a follow up question, how would this work if the schema ID is
> > not
> > > in
> > > > the Kafka headers, as hinted to in the FLIP "usually the global ID
> in a
> > > > Kafka header"? I am wondering if there are any other instances where
> > the
> > > > source would be aware of the schema ID and pass it through in this
> way?
> > > >
> > > > Thanks,
> > > > Danny
> > > >
> > > >
> > > >
> > > > On Wed, May 22, 2024 at 3:43 PM David Radley <
> david_radley@uk.ibm.com<ma...@uk.ibm.com>>
> > > > wrote:
> > > >
> > > > > Hi Danny,
> > > > > Did you have a chance you have a look at my responses to your
> > > feedback? I
> > > > > am hoping to keep the momentum going on this one,   kind regards,
> > > David.
> > > > >
> > > > >
> > > > > From: David Radley <david_radley@uk.ibm.com<mailto:
> david_radley@uk.ibm.com>>
> > > > > Date: Tuesday, 14 May 2024 at 17:21
> > > > > To: dev@flink.apache.org<ma...@flink.apache.org> <
> dev@flink.apache.org<ma...@flink.apache.org>>
> > > > > Subject: [EXTERNAL] [DISCUSS] FLIP-XXX Apicurio-avro format
> > > > > Hi Danny,
> > > > >
> > > > > Thank you very much for the feedback and your support. I have
> copied
> > > your
> > > > > feedback from the VOTE thread to this discussion thread, so we can
> > > > continue
> > > > > our discussions off the VOTE thread.
> > > > >
> > > > >
> > > > >
> > > > > Your feedback:
> > > > >
> > > > > Thanks for Driving this David. I am +1 for adding support for the
> new
> > > > >
> > > > > format, however have some questions/suggestions on the details.
> > > > >
> > > > >
> > > > >
> > > > > 1. Passing around Map<String, Object> additionalInputProperties
> > feels a
> > > > bit
> > > > >
> > > > > dirty. It looks like this is mainly for the Kafka connector. This
> > > > connector
> > > > >
> > > > > already has a de/serialization schema extension to access record
> > > > >
> > > > > headers, KafkaRecordDeserializationSchema [1], can we use this
> > instead?
> > > > >
> > > > > 2. Can you elaborate why we need to change the SchemaCoder
> interface?
> > > > Again
> > > > >
> > > > > I am not a fan of adding these Map parameters
> > > > >
> > > > > 3. I assume this integration will go into the core Flink repo under
> > > > >
> > > > > flink-formats [2], and not be a separate repository like the
> > > connectors?
> > > > >
> > > > >
> > > > >
> > > > > My response:
> > > > >
> > > > > Addressing 1. and 2.
> > > > >
> > > > > I agree that sending maps around is a bit dirty. If we can see a
> > better
> > > > > way that would be great. I was looking for a way to pass this kafka
> > > > header
> > > > > information in a non-Kafka way - the most obvious way I could think
> > was
> > > > as
> > > > > a map. Here are the main considerations I saw, if I have missed
> > > anything
> > > > or
> > > > > could improve something I would be grateful for any further
> feedback.
> > > > >
> > > > >
> > > > >
> > > > >   *   I see KafkaRecordDeserializationSchema is a Kafka interface
> > that
> > > > > works at the Kafka record level (so includes the headers). We need
> a
> > > > > mechanism to send over the headers from the Kafka record to Flink
> > > > >   *   Flink core is not aware of Kafka headers, and I did not want
> to
> > > add
> > > > > a Kafka dependancy to core flink.
> > > > >   *   The formats are stateless so it did not appear to be in
> fitting
> > > > with
> > > > > the Flink architecture to pass through header information to stash
> in
> > > > state
> > > > > in the format waiting for the deserialise to be subsequently called
> > to
> > > > pick
> > > > > up the header information.
> > > > >   *   We could have used Thread local storage to stash the header
> > > > content,
> > > > > but this would be extra state to manage; and this would seem like
> an
> > > > > obtrusive change.
> > > > >   *   The SchemaCoder deserialise is where Confluent Avro gets the
> > > schema
> > > > > id from the payload, so it can lookup the schema. In line with this
> > > > > approach it made sense to extend the deserialise so it had the
> header
> > > > > contents so the Apicurio Avro format could lookup the schema.
> > > > >   *   I did not want to have Apicurio specific logic in the Kafka
> > > > > connector, if we did we could pull out the appropriate headers and
> > only
> > > > > send over the schema ids.
> > > > >   *   For deserialise, the schema id we are interested in is the
> one
> > in
> > > > > the Kafka headers on the message and is for the writer schema (an
> > Avro
> > > > > format concept) currently used by the confluent-avro format in
> > > > deserialize.
> > > > >   *   For serialise the schema ids need to be obtained from
> apicurio
> > > then
> > > > > passed through to Kafka.
> > > > >   *   For serialise there is existing logic around handling the
> > > metadata
> > > > > which includes passing the headers. But the presence of the
> metadata
> > > > would
> > > > > imply we have a metadata column. Maybe a change to the metadata
> > > mechanism
> > > > > may have allowed to use to pass the headers, but not create a
> > metadata
> > > > > column; instead I pass through the additional headers in a map to
> be
> > > > > appended.
> > > > >
> > > > >
> > > > >
> > > > > 3.
> > > > >
> > > > > Yes this integration will go into the core Flink repo under
> > > > >
> > > > > flink-formats and sit next to the confluent-avro format. The Avro
> > > format
> > > > > has the concept of a Registry and drives the confluent-avro format.
> > The
> > > > > Apicurio Avro format will use the same approach.
> > > > >
> > > > > Unless otherwise stated above:
> > > > >
> > > > > IBM United Kingdom Limited
> > > > > Registered in England and Wales with number 741598
> > > > > Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6
> > 3AU
> > > > >
> > > > > Unless otherwise stated above:
> > > > >
> > > > > IBM United Kingdom Limited
> > > > > Registered in England and Wales with number 741598
> > > > > Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6
> > 3AU
> > > > >
> > > >
> > > > Unless otherwise stated above:
> > > >
> > > > IBM United Kingdom Limited
> > > > Registered in England and Wales with number 741598
> > > > Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6
> 3AU
> > > >
> > >
> > > Unless otherwise stated above:
> > >
> > > IBM United Kingdom Limited
> > > Registered in England and Wales with number 741598
> > > Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU
> > >
> > > Unless otherwise stated above:
> > >
> > > IBM United Kingdom Limited
> > > Registered in England and Wales with number 741598
> > > Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU
> > >
> >
> > Unless otherwise stated above:
> >
> > IBM United Kingdom Limited
> > Registered in England and Wales with number 741598
> > Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU
> > Unless otherwise stated above:
> >
> > IBM United Kingdom Limited
> > Registered in England and Wales with number 741598
> > Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU
> >
>
> Unless otherwise stated above:
>
> IBM United Kingdom Limited
> Registered in England and Wales with number 741598
> Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU
>

RE: [DISCUSS] FLIP-XXX Apicurio-avro format

Posted by David Radley <da...@uk.ibm.com>.
Hi Danny,
Thank you for your feedback. I have brought a fork up to date with one commit so you can
see the code; I need to test more scenarios and add more unit tests, but it shows the basic idea of the design, Chesnay and I agreed.

 Kind regards, David.


From: Danny Cranmer <da...@apache.org>
Date: Thursday, 25 July 2024 at 16:46
To: dev@flink.apache.org <de...@flink.apache.org>
Cc: David Radley <da...@uk.ibm.com>
Subject: [EXTERNAL] Re: [DISCUSS] FLIP-XXX Apicurio-avro format
Hey, Thanks for the updated FLIP. > 1. 20 or beyond Given this is intended to go in the Kafka connector we can target the next Kafka connector version (v3. 3. 0) for Flink 1. 18 and 1. 19 (or 1. 19 and 1. 20, if 1. 20 is released first). > RecordBasedSerialization: : public

Hey,

Thanks for the updated FLIP.

> 1.20 or beyond

Given this is intended to go in the Kafka connector we can target the next Kafka connector version (v3.3.0) for Flink 1.18 and 1.19 (or 1.19 and 1.20, if 1.20 is released first).


>> Agreed

 > RecordBasedSerialization::public boolean canProcess(byte[] customSerialisation);

Not sure what this is for, since we are passing a byte array and asking if it can be serialized? Can you elaborate how the Kafka connector interacts with these interfaces?
>> we are asking it can be processed. The canProcess has a default and also can be overridden. The idea is that if we are in another format and the apicurio jar is there then we might pick up the apicurio jar in the discovery. So we check that what we have discovered really can process the message. During trsting I have the key as raw and the value as apicurio, and saw the issue that necessitates this method.


> byte[] customSerialisation

There are a lot of byte arrays being passed around, I assume we are not doing heavy conversions multiple times? Is there scope to introduce a new intermediate type to prevent duplicate processing cycles?
Or even better, do you have a fork with these changes you can share?
>> https://github.com/davidradl/flink-connector-kafka/tree/FLINK-35311-3

> Format 'avro-apicurio'

Since this is Kafka only I would be tempted to include Kafka in this tag, just in case we decide to add a Kafka agnostic format in the future. Additionally this makes it clearer to the user it is Kafka only.
>> I was keeping this in line with the avro-confluent naming which is also Kafka only. I do not have strong views on this – so I can change if we think it is better.

Thanks,
Danny



On Thu, Jul 18, 2024 at 6:32 PM Martijn Visser <ma...@apache.org>> wrote:
Hi David,

The FLIP is updated.

Cheers, Martijn


On Tue, Jul 16, 2024 at 6:33 PM David Radley <da...@uk.ibm.com>>
wrote:

> Hello all,
>
>
>
> I have prototyped the new design and confirmed it works. I have documented
> the design in google doc
> https://docs.google.com/document/d/1J1E-cE-X2H3-kw4rNjLn71OGPQk_Yl1iGX4-eCHWLgE/edit
>
>
>
> By copy Martijn: please could you move this content over to replace the
> content of
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-454%3A+New+Apicurio+Avro+format
>
>
>
> As the design is significantly different, there will be a new period of
> discussion before going to vote.
>
>
>
> Fyi I will be on vacation after this week until the 13th of August.
>
>
>
> Kind regards, David.
>
>
>
>
>
>
>
> *From: *David Radley <da...@uk.ibm.com>>
> *Date: *Tuesday, 9 July 2024 at 11:17
> *To: *dev@flink.apache.org<ma...@flink.apache.org> <de...@flink.apache.org>>
> *Subject: *[EXTERNAL] RE: [DISCUSS] FLIP-XXX Apicurio-avro format
>
> Hi Kevin,
> I have agreed a design with Chesnay and Danny. I am implementing a
> prototype, to prove it works,  then will update the Flip text with the new
> design. Initial testing is showing it working.
>
> Here is a quick history so you can understand our current thinking.
>
>   1.  Initially we passed maps for header information from the kafka
> connector to Flink for deserialization. Similar for serialize. This was not
> great, because maps are not ideal and it was a big change as it needed core
> Flink interface changes
>   2.  We then moved the Avro Apicurio format to the Kafka connector and
> looked to discover a new record based de/serialization interface. So we
> could pass down the record (containing the headers) rather than the
> payload. This did not work, because there is a dependence on the Avro
> connector that is not  aware of the new interface.
>   3.  We considered using Thread local storage to pass the headers, we did
> not like this as there was a risk of memory leaks if we did not manage the
> thread well, also the contract is hidden.
>   4.  We then came up with the current design that augments the
> deserialization in the Kafka connector in a new discovered record based
> deserialization, it then takes the headers out in the schema coder, leaving
> the message as it was. Similar for serialization.
>
>
> One piece I need to work out the details of, is how to work when there are
> 2 implementations that can be discovered, probably using an augmented
> format name as a factory identifier,
>
> I hope to put up a new design in the Flip by the end of next week, for
> wider review,
>     Kind regards, David.
>
>
> From: Kevin Lam <ke...@shopify.com.INVALID>
> Date: Monday, 8 July 2024 at 21:16
> To: dev@flink.apache.org<ma...@flink.apache.org> <de...@flink.apache.org>>
> Subject: [EXTERNAL] Re: [DISCUSS] FLIP-XXX Apicurio-avro format
> Hi David,
>
> Any updates on the Kafka Message Header support? I am also interested in
> supporting headers with the Flink SQL Formats:
> https://lists.apache.org/thread/spl88o63sjm2dv4l5no0ym632d2yt2o6
>
> On Fri, Jun 14, 2024 at 6:10 AM David Radley <da...@uk.ibm.com>>
> wrote:
>
> > Hi everyone,
> > I have talked with Chesnay and Danny offline. Danny and I were not very
> > happy with the passing Maps around, and were looking for a neater design.
> > Chesnay suggested that we could move the new format to the Kafka
> connector,
> > then pass the Kafka record down to the deserialize logic so it can make
> use
> > of the headers during deserialization and serialisation.
> >
> > I think this is a neat idea. This would mean:
> > - the Kafka connector code would need to be updated to pass down the
> Kafka
> > record
> > - there would be the Avro Apicurio format and SQL in the kafka
> repository.
> > We feel it is unlikely to want to use the Apicurio registry with files,
> as
> > the Avro format could be used.
> >
> > Unfortunately I have found that this as not so straight forward to
> > implement as the Avro Apicurio format uses the Avro format, which is tied
> > to the DeserializationSchema. We were hoping to have a new decoding
> > implementation that would pass down the Kafka record rather than the
> > payload. This does not appear possible without a Avro format change.
> >
> >
> > Inspired by this idea, I notice that
> > KafkaValueOnlyRecordDeserializerWrapper<T> extends
> > KafkaValueOnlyDeserializerWrapper
> >
> > Does
> >
> > deserializer.deserialize(record.topic(),record.value())
> >
> >
> >
> > I am investigating If I can add a factory/reflection to provide an
> > alternative
> > Implementation that will pass the record based (the kafka record is not
> > serializable so I will pick what we need and deserialize) as a byte
> array.
> >
> > I would need to do this 4 times (value ,key for deserialisation and
> > serialisation. To do this I would need to convert the record into a byte
> > array, so it fits into the existing interface (DeserializationSchema).  I
> > think this could be a way through, to avoid using maps and avoid changing
> > the existing Avro format and avoid change any core Flink interfaces.
> >
> > I am going to prototype this idea. WDYT?
> >
> > My thanks go to Chesnay and Danny for their support and insight around
> > this Flip,
> >    Kind regards, David.
> >
> >
> >
> >
> >
> >
> > From: David Radley <da...@uk.ibm.com>>
> > Date: Wednesday, 29 May 2024 at 11:39
> > To: dev@flink.apache.org<ma...@flink.apache.org> <de...@flink.apache.org>>
> > Subject: [EXTERNAL] RE: [DISCUSS] FLIP-XXX Apicurio-avro format
> > Hi Danny,
> > Thank you for your feedback on this.
> >
> > I agree that using maps has pros and cons. The maps are flexible, but do
> > require the sender and receiver to know what is in the map.
> >
> > When you say “That sounds like it would fit in better, I assume we cannot
> > just take that approach?” The motivation behind this Flip is to support
> the
> > headers which is the usual way that Apicurio runs. We will support the
> > “schema id in the payload” as well.
> >
> > I agree with you when you say “ I am not 100% happy with the solution
> but I
> > cannot offer a better option.” – this is a pragmatic way we have found to
> > solve this issue. I am open to any suggestions to improve this as well.
> >
> > If we are going with the maps design (which is the best we have at the
> > moment) ; it would be good to have the Flink core changes in base Flink
> > version 2.0 as this would mean we do not need to use reflection in a
> Flink
> > Kafka version 2 connector to work out if the runtime Flink has the new
> > methods.
> >
> > At this stage we only have one committer (yourself) backing this. Do you
> > know of other 2 committers who would support this Flip?
> >
> >      Kind regards, David.
> >
> >
> >
> > From: Danny Cranmer <da...@apache.org>>
> > Date: Friday, 24 May 2024 at 19:32
> > To: dev@flink.apache.org<ma...@flink.apache.org> <de...@flink.apache.org>>
> > Subject: [EXTERNAL] Re: [DISCUSS] FLIP-XXX Apicurio-avro format
> > Hello,
> >
> > > I am curious what you mean by abused.
> >
> > I just meant we will end up adding more and more fields to this map over
> > time, and it may be hard to undo.
> >
> > > For Apicurio it can be sent at the start of the payload like Confluent
> > Avro does. Confluent Avro have a magic byte followed by 4 bytes of schema
> > id, at the start of the payload. Apicurio clients and SerDe libraries can
> > be configured to not put the schema id in the headers in which case there
> > is a magic byte followed by an 8 byte schema at the start of the payload.
> > In the deserialization case, we would not need to look at the headers –
> > though the encoding is also in the headers.
> >
> > That sounds like it would fit in better, I assume we cannot just take
> that
> > approach?
> >
> > Thanks for the discussion. I am not 100% happy with the solution but I
> > cannot offer a better option. I would be interested to hear if others
> have
> > any suggestions. Playing devil's advocate against myself, we pass maps
> > around to configure connectors so it is not too far away from that.
> >
> > Thanks,
> > Danny
> >
> >
> > On Fri, May 24, 2024 at 2:23 PM David Radley <da...@uk.ibm.com>>
> > wrote:
> >
> > > Hi Danny,
> > > No worries, thanks for replying. I have working prototype code that is
> > > being reviewed. It needs some cleaning up and more complete testing
> > before
> > > it is ready, but will give you the general idea [1][2] to help to
> assess
> > > this approach.
> > >
> > >
> > > I am curious what you mean by abused. I guess the approaches are
> between
> > > generic map, mechanism vs a more particular more granular things being
> > > passed that might be used by another connector.
> > >
> > > Your first question:
> > > “how would this work if the schema ID is not in the Kafka headers, as
> > > hinted to in the FLIP "usually the global ID in a Kafka header"?
> > >
> > > For Apicurio it can be sent at the start of the payload like Confluent
> > > Avro does. Confluent Avro have a magic byte followed by 4 bytes of
> schema
> > > id, at the start of the payload. Apicurio clients and SerDe libraries
> can
> > > be configured to not put the schema id in the headers in which case
> there
> > > is a magic byte followed by an 8 byte schema at the start of the
> payload.
> > > In the deserialization case, we would not need to look at the headers –
> > > though the encoding is also in the headers.
> > >
> > > Your second question:
> > > “I am wondering if there are any other instances where the source would
> > be
> > > aware of the schema ID and pass it through in this way?
> > > ”
> > > The examples I can think of are:
> > > - Avro can send the complete schema in a header, this is not
> recommended
> > > but in theory fits the need for a message payload to require something
> > else
> > > to get the structure.
> > > - I see [2] that Apicurio Protobuf uses headers.
> > > - it might be that other message queuing projects like Rabbit MQ would
> > > need this to be able to support Apicurio Avro & protobuf.
> > >
> > > Kind regards, David,
> > >
> > >
> > >
> > >
> > > [1] https://github.com/apache/flink/pull/24715
> > > [2] https://github.com/apache/flink-connector-kafka/pull/99
> > > [3]
> > >
> >
> https://www.apicur.io/registry/docs/apicurio-registry/2.5.x/getting-started/assembly-configuring-kafka-client-serdes.html#registry-serdes-types-json_registry
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > From: Danny Cranmer <da...@apache.org>>
> > > Date: Friday, 24 May 2024 at 12:22
> > > To: dev@flink.apache.org<ma...@flink.apache.org> <de...@flink.apache.org>>
> > > Subject: [EXTERNAL] Re: [DISCUSS] FLIP-XXX Apicurio-avro format
> > > Hello,
> > >
> > > Apologies, I am on vacation and have limited access to email.
> > >
> > > I can see the logic here and why you ended up where you did. I can also
> > see
> > > there are other useful metadata fields that we might want to pass
> > through,
> > > which might result in this Map being abused (Kafka Topic, Kinesis
> Shard,
> > > etc).
> > >
> > > I have a follow up question, how would this work if the schema ID is
> not
> > in
> > > the Kafka headers, as hinted to in the FLIP "usually the global ID in a
> > > Kafka header"? I am wondering if there are any other instances where
> the
> > > source would be aware of the schema ID and pass it through in this way?
> > >
> > > Thanks,
> > > Danny
> > >
> > >
> > >
> > > On Wed, May 22, 2024 at 3:43 PM David Radley <da...@uk.ibm.com>>
> > > wrote:
> > >
> > > > Hi Danny,
> > > > Did you have a chance you have a look at my responses to your
> > feedback? I
> > > > am hoping to keep the momentum going on this one,   kind regards,
> > David.
> > > >
> > > >
> > > > From: David Radley <da...@uk.ibm.com>>
> > > > Date: Tuesday, 14 May 2024 at 17:21
> > > > To: dev@flink.apache.org<ma...@flink.apache.org> <de...@flink.apache.org>>
> > > > Subject: [EXTERNAL] [DISCUSS] FLIP-XXX Apicurio-avro format
> > > > Hi Danny,
> > > >
> > > > Thank you very much for the feedback and your support. I have copied
> > your
> > > > feedback from the VOTE thread to this discussion thread, so we can
> > > continue
> > > > our discussions off the VOTE thread.
> > > >
> > > >
> > > >
> > > > Your feedback:
> > > >
> > > > Thanks for Driving this David. I am +1 for adding support for the new
> > > >
> > > > format, however have some questions/suggestions on the details.
> > > >
> > > >
> > > >
> > > > 1. Passing around Map<String, Object> additionalInputProperties
> feels a
> > > bit
> > > >
> > > > dirty. It looks like this is mainly for the Kafka connector. This
> > > connector
> > > >
> > > > already has a de/serialization schema extension to access record
> > > >
> > > > headers, KafkaRecordDeserializationSchema [1], can we use this
> instead?
> > > >
> > > > 2. Can you elaborate why we need to change the SchemaCoder interface?
> > > Again
> > > >
> > > > I am not a fan of adding these Map parameters
> > > >
> > > > 3. I assume this integration will go into the core Flink repo under
> > > >
> > > > flink-formats [2], and not be a separate repository like the
> > connectors?
> > > >
> > > >
> > > >
> > > > My response:
> > > >
> > > > Addressing 1. and 2.
> > > >
> > > > I agree that sending maps around is a bit dirty. If we can see a
> better
> > > > way that would be great. I was looking for a way to pass this kafka
> > > header
> > > > information in a non-Kafka way - the most obvious way I could think
> was
> > > as
> > > > a map. Here are the main considerations I saw, if I have missed
> > anything
> > > or
> > > > could improve something I would be grateful for any further feedback.
> > > >
> > > >
> > > >
> > > >   *   I see KafkaRecordDeserializationSchema is a Kafka interface
> that
> > > > works at the Kafka record level (so includes the headers). We need a
> > > > mechanism to send over the headers from the Kafka record to Flink
> > > >   *   Flink core is not aware of Kafka headers, and I did not want to
> > add
> > > > a Kafka dependancy to core flink.
> > > >   *   The formats are stateless so it did not appear to be in fitting
> > > with
> > > > the Flink architecture to pass through header information to stash in
> > > state
> > > > in the format waiting for the deserialise to be subsequently called
> to
> > > pick
> > > > up the header information.
> > > >   *   We could have used Thread local storage to stash the header
> > > content,
> > > > but this would be extra state to manage; and this would seem like an
> > > > obtrusive change.
> > > >   *   The SchemaCoder deserialise is where Confluent Avro gets the
> > schema
> > > > id from the payload, so it can lookup the schema. In line with this
> > > > approach it made sense to extend the deserialise so it had the header
> > > > contents so the Apicurio Avro format could lookup the schema.
> > > >   *   I did not want to have Apicurio specific logic in the Kafka
> > > > connector, if we did we could pull out the appropriate headers and
> only
> > > > send over the schema ids.
> > > >   *   For deserialise, the schema id we are interested in is the one
> in
> > > > the Kafka headers on the message and is for the writer schema (an
> Avro
> > > > format concept) currently used by the confluent-avro format in
> > > deserialize.
> > > >   *   For serialise the schema ids need to be obtained from apicurio
> > then
> > > > passed through to Kafka.
> > > >   *   For serialise there is existing logic around handling the
> > metadata
> > > > which includes passing the headers. But the presence of the metadata
> > > would
> > > > imply we have a metadata column. Maybe a change to the metadata
> > mechanism
> > > > may have allowed to use to pass the headers, but not create a
> metadata
> > > > column; instead I pass through the additional headers in a map to be
> > > > appended.
> > > >
> > > >
> > > >
> > > > 3.
> > > >
> > > > Yes this integration will go into the core Flink repo under
> > > >
> > > > flink-formats and sit next to the confluent-avro format. The Avro
> > format
> > > > has the concept of a Registry and drives the confluent-avro format.
> The
> > > > Apicurio Avro format will use the same approach.
> > > >
> > > > Unless otherwise stated above:
> > > >
> > > > IBM United Kingdom Limited
> > > > Registered in England and Wales with number 741598
> > > > Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6
> 3AU
> > > >
> > > > Unless otherwise stated above:
> > > >
> > > > IBM United Kingdom Limited
> > > > Registered in England and Wales with number 741598
> > > > Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6
> 3AU
> > > >
> > >
> > > Unless otherwise stated above:
> > >
> > > IBM United Kingdom Limited
> > > Registered in England and Wales with number 741598
> > > Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU
> > >
> >
> > Unless otherwise stated above:
> >
> > IBM United Kingdom Limited
> > Registered in England and Wales with number 741598
> > Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU
> >
> > Unless otherwise stated above:
> >
> > IBM United Kingdom Limited
> > Registered in England and Wales with number 741598
> > Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU
> >
>
> Unless otherwise stated above:
>
> IBM United Kingdom Limited
> Registered in England and Wales with number 741598
> Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU
> Unless otherwise stated above:
>
> IBM United Kingdom Limited
> Registered in England and Wales with number 741598
> Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU
>

Unless otherwise stated above:

IBM United Kingdom Limited
Registered in England and Wales with number 741598
Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU

Re: [DISCUSS] FLIP-XXX Apicurio-avro format

Posted by Danny Cranmer <da...@apache.org>.
Hey,

Thanks for the updated FLIP.

> 1.20 or beyond

Given this is intended to go in the Kafka connector we can target the next
Kafka connector version (v3.3.0) for Flink 1.18 and 1.19 (or 1.19 and 1.20,
if 1.20 is released first).

 > RecordBasedSerialization::public boolean canProcess(byte[]
customSerialisation);

Not sure what this is for, since we are passing a byte array and asking if
it can be serialized? Can you elaborate how the Kafka connector interacts
with these interfaces?

> byte[] customSerialisation

There are a lot of byte arrays being passed around, I assume we are not
doing heavy conversions multiple times? Is there scope to introduce a new
intermediate type to prevent duplicate processing cycles?
Or even better, do you have a fork with these changes you can share?

> Format 'avro-apicurio'

Since this is Kafka only I would be tempted to include Kafka in this tag,
just in case we decide to add a Kafka agnostic format in the future.
Additionally this makes it clearer to the user it is Kafka only.

Thanks,
Danny



On Thu, Jul 18, 2024 at 6:32 PM Martijn Visser <ma...@apache.org>
wrote:

> Hi David,
>
> The FLIP is updated.
>
> Cheers, Martijn
>
>
> On Tue, Jul 16, 2024 at 6:33 PM David Radley <da...@uk.ibm.com>
> wrote:
>
> > Hello all,
> >
> >
> >
> > I have prototyped the new design and confirmed it works. I have
> documented
> > the design in google doc
> >
> https://docs.google.com/document/d/1J1E-cE-X2H3-kw4rNjLn71OGPQk_Yl1iGX4-eCHWLgE/edit
> >
> >
> >
> > By copy Martijn: please could you move this content over to replace the
> > content of
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-454%3A+New+Apicurio+Avro+format
> >
> >
> >
> > As the design is significantly different, there will be a new period of
> > discussion before going to vote.
> >
> >
> >
> > Fyi I will be on vacation after this week until the 13th of August.
> >
> >
> >
> > Kind regards, David.
> >
> >
> >
> >
> >
> >
> >
> > *From: *David Radley <da...@uk.ibm.com>
> > *Date: *Tuesday, 9 July 2024 at 11:17
> > *To: *dev@flink.apache.org <de...@flink.apache.org>
> > *Subject: *[EXTERNAL] RE: [DISCUSS] FLIP-XXX Apicurio-avro format
> >
> > Hi Kevin,
> > I have agreed a design with Chesnay and Danny. I am implementing a
> > prototype, to prove it works,  then will update the Flip text with the
> new
> > design. Initial testing is showing it working.
> >
> > Here is a quick history so you can understand our current thinking.
> >
> >   1.  Initially we passed maps for header information from the kafka
> > connector to Flink for deserialization. Similar for serialize. This was
> not
> > great, because maps are not ideal and it was a big change as it needed
> core
> > Flink interface changes
> >   2.  We then moved the Avro Apicurio format to the Kafka connector and
> > looked to discover a new record based de/serialization interface. So we
> > could pass down the record (containing the headers) rather than the
> > payload. This did not work, because there is a dependence on the Avro
> > connector that is not  aware of the new interface.
> >   3.  We considered using Thread local storage to pass the headers, we
> did
> > not like this as there was a risk of memory leaks if we did not manage
> the
> > thread well, also the contract is hidden.
> >   4.  We then came up with the current design that augments the
> > deserialization in the Kafka connector in a new discovered record based
> > deserialization, it then takes the headers out in the schema coder,
> leaving
> > the message as it was. Similar for serialization.
> >
> >
> > One piece I need to work out the details of, is how to work when there
> are
> > 2 implementations that can be discovered, probably using an augmented
> > format name as a factory identifier,
> >
> > I hope to put up a new design in the Flip by the end of next week, for
> > wider review,
> >     Kind regards, David.
> >
> >
> > From: Kevin Lam <ke...@shopify.com.INVALID>
> > Date: Monday, 8 July 2024 at 21:16
> > To: dev@flink.apache.org <de...@flink.apache.org>
> > Subject: [EXTERNAL] Re: [DISCUSS] FLIP-XXX Apicurio-avro format
> > Hi David,
> >
> > Any updates on the Kafka Message Header support? I am also interested in
> > supporting headers with the Flink SQL Formats:
> > https://lists.apache.org/thread/spl88o63sjm2dv4l5no0ym632d2yt2o6
> >
> > On Fri, Jun 14, 2024 at 6:10 AM David Radley <da...@uk.ibm.com>
> > wrote:
> >
> > > Hi everyone,
> > > I have talked with Chesnay and Danny offline. Danny and I were not very
> > > happy with the passing Maps around, and were looking for a neater
> design.
> > > Chesnay suggested that we could move the new format to the Kafka
> > connector,
> > > then pass the Kafka record down to the deserialize logic so it can make
> > use
> > > of the headers during deserialization and serialisation.
> > >
> > > I think this is a neat idea. This would mean:
> > > - the Kafka connector code would need to be updated to pass down the
> > Kafka
> > > record
> > > - there would be the Avro Apicurio format and SQL in the kafka
> > repository.
> > > We feel it is unlikely to want to use the Apicurio registry with files,
> > as
> > > the Avro format could be used.
> > >
> > > Unfortunately I have found that this as not so straight forward to
> > > implement as the Avro Apicurio format uses the Avro format, which is
> tied
> > > to the DeserializationSchema. We were hoping to have a new decoding
> > > implementation that would pass down the Kafka record rather than the
> > > payload. This does not appear possible without a Avro format change.
> > >
> > >
> > > Inspired by this idea, I notice that
> > > KafkaValueOnlyRecordDeserializerWrapper<T> extends
> > > KafkaValueOnlyDeserializerWrapper
> > >
> > > Does
> > >
> > > deserializer.deserialize(record.topic(),record.value())
> > >
> > >
> > >
> > > I am investigating If I can add a factory/reflection to provide an
> > > alternative
> > > Implementation that will pass the record based (the kafka record is not
> > > serializable so I will pick what we need and deserialize) as a byte
> > array.
> > >
> > > I would need to do this 4 times (value ,key for deserialisation and
> > > serialisation. To do this I would need to convert the record into a
> byte
> > > array, so it fits into the existing interface
> (DeserializationSchema).  I
> > > think this could be a way through, to avoid using maps and avoid
> changing
> > > the existing Avro format and avoid change any core Flink interfaces.
> > >
> > > I am going to prototype this idea. WDYT?
> > >
> > > My thanks go to Chesnay and Danny for their support and insight around
> > > this Flip,
> > >    Kind regards, David.
> > >
> > >
> > >
> > >
> > >
> > >
> > > From: David Radley <da...@uk.ibm.com>
> > > Date: Wednesday, 29 May 2024 at 11:39
> > > To: dev@flink.apache.org <de...@flink.apache.org>
> > > Subject: [EXTERNAL] RE: [DISCUSS] FLIP-XXX Apicurio-avro format
> > > Hi Danny,
> > > Thank you for your feedback on this.
> > >
> > > I agree that using maps has pros and cons. The maps are flexible, but
> do
> > > require the sender and receiver to know what is in the map.
> > >
> > > When you say “That sounds like it would fit in better, I assume we
> cannot
> > > just take that approach?” The motivation behind this Flip is to support
> > the
> > > headers which is the usual way that Apicurio runs. We will support the
> > > “schema id in the payload” as well.
> > >
> > > I agree with you when you say “ I am not 100% happy with the solution
> > but I
> > > cannot offer a better option.” – this is a pragmatic way we have found
> to
> > > solve this issue. I am open to any suggestions to improve this as well.
> > >
> > > If we are going with the maps design (which is the best we have at the
> > > moment) ; it would be good to have the Flink core changes in base Flink
> > > version 2.0 as this would mean we do not need to use reflection in a
> > Flink
> > > Kafka version 2 connector to work out if the runtime Flink has the new
> > > methods.
> > >
> > > At this stage we only have one committer (yourself) backing this. Do
> you
> > > know of other 2 committers who would support this Flip?
> > >
> > >      Kind regards, David.
> > >
> > >
> > >
> > > From: Danny Cranmer <da...@apache.org>
> > > Date: Friday, 24 May 2024 at 19:32
> > > To: dev@flink.apache.org <de...@flink.apache.org>
> > > Subject: [EXTERNAL] Re: [DISCUSS] FLIP-XXX Apicurio-avro format
> > > Hello,
> > >
> > > > I am curious what you mean by abused.
> > >
> > > I just meant we will end up adding more and more fields to this map
> over
> > > time, and it may be hard to undo.
> > >
> > > > For Apicurio it can be sent at the start of the payload like
> Confluent
> > > Avro does. Confluent Avro have a magic byte followed by 4 bytes of
> schema
> > > id, at the start of the payload. Apicurio clients and SerDe libraries
> can
> > > be configured to not put the schema id in the headers in which case
> there
> > > is a magic byte followed by an 8 byte schema at the start of the
> payload.
> > > In the deserialization case, we would not need to look at the headers –
> > > though the encoding is also in the headers.
> > >
> > > That sounds like it would fit in better, I assume we cannot just take
> > that
> > > approach?
> > >
> > > Thanks for the discussion. I am not 100% happy with the solution but I
> > > cannot offer a better option. I would be interested to hear if others
> > have
> > > any suggestions. Playing devil's advocate against myself, we pass maps
> > > around to configure connectors so it is not too far away from that.
> > >
> > > Thanks,
> > > Danny
> > >
> > >
> > > On Fri, May 24, 2024 at 2:23 PM David Radley <da...@uk.ibm.com>
> > > wrote:
> > >
> > > > Hi Danny,
> > > > No worries, thanks for replying. I have working prototype code that
> is
> > > > being reviewed. It needs some cleaning up and more complete testing
> > > before
> > > > it is ready, but will give you the general idea [1][2] to help to
> > assess
> > > > this approach.
> > > >
> > > >
> > > > I am curious what you mean by abused. I guess the approaches are
> > between
> > > > generic map, mechanism vs a more particular more granular things
> being
> > > > passed that might be used by another connector.
> > > >
> > > > Your first question:
> > > > “how would this work if the schema ID is not in the Kafka headers, as
> > > > hinted to in the FLIP "usually the global ID in a Kafka header"?
> > > >
> > > > For Apicurio it can be sent at the start of the payload like
> Confluent
> > > > Avro does. Confluent Avro have a magic byte followed by 4 bytes of
> > schema
> > > > id, at the start of the payload. Apicurio clients and SerDe libraries
> > can
> > > > be configured to not put the schema id in the headers in which case
> > there
> > > > is a magic byte followed by an 8 byte schema at the start of the
> > payload.
> > > > In the deserialization case, we would not need to look at the
> headers –
> > > > though the encoding is also in the headers.
> > > >
> > > > Your second question:
> > > > “I am wondering if there are any other instances where the source
> would
> > > be
> > > > aware of the schema ID and pass it through in this way?
> > > > ”
> > > > The examples I can think of are:
> > > > - Avro can send the complete schema in a header, this is not
> > recommended
> > > > but in theory fits the need for a message payload to require
> something
> > > else
> > > > to get the structure.
> > > > - I see [2] that Apicurio Protobuf uses headers.
> > > > - it might be that other message queuing projects like Rabbit MQ
> would
> > > > need this to be able to support Apicurio Avro & protobuf.
> > > >
> > > > Kind regards, David,
> > > >
> > > >
> > > >
> > > >
> > > > [1] https://github.com/apache/flink/pull/24715
> > > > [2] https://github.com/apache/flink-connector-kafka/pull/99
> > > > [3]
> > > >
> > >
> >
> https://www.apicur.io/registry/docs/apicurio-registry/2.5.x/getting-started/assembly-configuring-kafka-client-serdes.html#registry-serdes-types-json_registry
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > From: Danny Cranmer <da...@apache.org>
> > > > Date: Friday, 24 May 2024 at 12:22
> > > > To: dev@flink.apache.org <de...@flink.apache.org>
> > > > Subject: [EXTERNAL] Re: [DISCUSS] FLIP-XXX Apicurio-avro format
> > > > Hello,
> > > >
> > > > Apologies, I am on vacation and have limited access to email.
> > > >
> > > > I can see the logic here and why you ended up where you did. I can
> also
> > > see
> > > > there are other useful metadata fields that we might want to pass
> > > through,
> > > > which might result in this Map being abused (Kafka Topic, Kinesis
> > Shard,
> > > > etc).
> > > >
> > > > I have a follow up question, how would this work if the schema ID is
> > not
> > > in
> > > > the Kafka headers, as hinted to in the FLIP "usually the global ID
> in a
> > > > Kafka header"? I am wondering if there are any other instances where
> > the
> > > > source would be aware of the schema ID and pass it through in this
> way?
> > > >
> > > > Thanks,
> > > > Danny
> > > >
> > > >
> > > >
> > > > On Wed, May 22, 2024 at 3:43 PM David Radley <
> david_radley@uk.ibm.com>
> > > > wrote:
> > > >
> > > > > Hi Danny,
> > > > > Did you have a chance you have a look at my responses to your
> > > feedback? I
> > > > > am hoping to keep the momentum going on this one,   kind regards,
> > > David.
> > > > >
> > > > >
> > > > > From: David Radley <da...@uk.ibm.com>
> > > > > Date: Tuesday, 14 May 2024 at 17:21
> > > > > To: dev@flink.apache.org <de...@flink.apache.org>
> > > > > Subject: [EXTERNAL] [DISCUSS] FLIP-XXX Apicurio-avro format
> > > > > Hi Danny,
> > > > >
> > > > > Thank you very much for the feedback and your support. I have
> copied
> > > your
> > > > > feedback from the VOTE thread to this discussion thread, so we can
> > > > continue
> > > > > our discussions off the VOTE thread.
> > > > >
> > > > >
> > > > >
> > > > > Your feedback:
> > > > >
> > > > > Thanks for Driving this David. I am +1 for adding support for the
> new
> > > > >
> > > > > format, however have some questions/suggestions on the details.
> > > > >
> > > > >
> > > > >
> > > > > 1. Passing around Map<String, Object> additionalInputProperties
> > feels a
> > > > bit
> > > > >
> > > > > dirty. It looks like this is mainly for the Kafka connector. This
> > > > connector
> > > > >
> > > > > already has a de/serialization schema extension to access record
> > > > >
> > > > > headers, KafkaRecordDeserializationSchema [1], can we use this
> > instead?
> > > > >
> > > > > 2. Can you elaborate why we need to change the SchemaCoder
> interface?
> > > > Again
> > > > >
> > > > > I am not a fan of adding these Map parameters
> > > > >
> > > > > 3. I assume this integration will go into the core Flink repo under
> > > > >
> > > > > flink-formats [2], and not be a separate repository like the
> > > connectors?
> > > > >
> > > > >
> > > > >
> > > > > My response:
> > > > >
> > > > > Addressing 1. and 2.
> > > > >
> > > > > I agree that sending maps around is a bit dirty. If we can see a
> > better
> > > > > way that would be great. I was looking for a way to pass this kafka
> > > > header
> > > > > information in a non-Kafka way - the most obvious way I could think
> > was
> > > > as
> > > > > a map. Here are the main considerations I saw, if I have missed
> > > anything
> > > > or
> > > > > could improve something I would be grateful for any further
> feedback.
> > > > >
> > > > >
> > > > >
> > > > >   *   I see KafkaRecordDeserializationSchema is a Kafka interface
> > that
> > > > > works at the Kafka record level (so includes the headers). We need
> a
> > > > > mechanism to send over the headers from the Kafka record to Flink
> > > > >   *   Flink core is not aware of Kafka headers, and I did not want
> to
> > > add
> > > > > a Kafka dependancy to core flink.
> > > > >   *   The formats are stateless so it did not appear to be in
> fitting
> > > > with
> > > > > the Flink architecture to pass through header information to stash
> in
> > > > state
> > > > > in the format waiting for the deserialise to be subsequently called
> > to
> > > > pick
> > > > > up the header information.
> > > > >   *   We could have used Thread local storage to stash the header
> > > > content,
> > > > > but this would be extra state to manage; and this would seem like
> an
> > > > > obtrusive change.
> > > > >   *   The SchemaCoder deserialise is where Confluent Avro gets the
> > > schema
> > > > > id from the payload, so it can lookup the schema. In line with this
> > > > > approach it made sense to extend the deserialise so it had the
> header
> > > > > contents so the Apicurio Avro format could lookup the schema.
> > > > >   *   I did not want to have Apicurio specific logic in the Kafka
> > > > > connector, if we did we could pull out the appropriate headers and
> > only
> > > > > send over the schema ids.
> > > > >   *   For deserialise, the schema id we are interested in is the
> one
> > in
> > > > > the Kafka headers on the message and is for the writer schema (an
> > Avro
> > > > > format concept) currently used by the confluent-avro format in
> > > > deserialize.
> > > > >   *   For serialise the schema ids need to be obtained from
> apicurio
> > > then
> > > > > passed through to Kafka.
> > > > >   *   For serialise there is existing logic around handling the
> > > metadata
> > > > > which includes passing the headers. But the presence of the
> metadata
> > > > would
> > > > > imply we have a metadata column. Maybe a change to the metadata
> > > mechanism
> > > > > may have allowed to use to pass the headers, but not create a
> > metadata
> > > > > column; instead I pass through the additional headers in a map to
> be
> > > > > appended.
> > > > >
> > > > >
> > > > >
> > > > > 3.
> > > > >
> > > > > Yes this integration will go into the core Flink repo under
> > > > >
> > > > > flink-formats and sit next to the confluent-avro format. The Avro
> > > format
> > > > > has the concept of a Registry and drives the confluent-avro format.
> > The
> > > > > Apicurio Avro format will use the same approach.
> > > > >
> > > > > Unless otherwise stated above:
> > > > >
> > > > > IBM United Kingdom Limited
> > > > > Registered in England and Wales with number 741598
> > > > > Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6
> > 3AU
> > > > >
> > > > > Unless otherwise stated above:
> > > > >
> > > > > IBM United Kingdom Limited
> > > > > Registered in England and Wales with number 741598
> > > > > Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6
> > 3AU
> > > > >
> > > >
> > > > Unless otherwise stated above:
> > > >
> > > > IBM United Kingdom Limited
> > > > Registered in England and Wales with number 741598
> > > > Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6
> 3AU
> > > >
> > >
> > > Unless otherwise stated above:
> > >
> > > IBM United Kingdom Limited
> > > Registered in England and Wales with number 741598
> > > Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU
> > >
> > > Unless otherwise stated above:
> > >
> > > IBM United Kingdom Limited
> > > Registered in England and Wales with number 741598
> > > Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU
> > >
> >
> > Unless otherwise stated above:
> >
> > IBM United Kingdom Limited
> > Registered in England and Wales with number 741598
> > Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU
> > Unless otherwise stated above:
> >
> > IBM United Kingdom Limited
> > Registered in England and Wales with number 741598
> > Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU
> >
>

Re: [DISCUSS] FLIP-XXX Apicurio-avro format

Posted by Martijn Visser <ma...@apache.org>.
Hi David,

The FLIP is updated.

Cheers, Martijn


On Tue, Jul 16, 2024 at 6:33 PM David Radley <da...@uk.ibm.com>
wrote:

> Hello all,
>
>
>
> I have prototyped the new design and confirmed it works. I have documented
> the design in google doc
> https://docs.google.com/document/d/1J1E-cE-X2H3-kw4rNjLn71OGPQk_Yl1iGX4-eCHWLgE/edit
>
>
>
> By copy Martijn: please could you move this content over to replace the
> content of
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-454%3A+New+Apicurio+Avro+format
>
>
>
> As the design is significantly different, there will be a new period of
> discussion before going to vote.
>
>
>
> Fyi I will be on vacation after this week until the 13th of August.
>
>
>
> Kind regards, David.
>
>
>
>
>
>
>
> *From: *David Radley <da...@uk.ibm.com>
> *Date: *Tuesday, 9 July 2024 at 11:17
> *To: *dev@flink.apache.org <de...@flink.apache.org>
> *Subject: *[EXTERNAL] RE: [DISCUSS] FLIP-XXX Apicurio-avro format
>
> Hi Kevin,
> I have agreed a design with Chesnay and Danny. I am implementing a
> prototype, to prove it works,  then will update the Flip text with the new
> design. Initial testing is showing it working.
>
> Here is a quick history so you can understand our current thinking.
>
>   1.  Initially we passed maps for header information from the kafka
> connector to Flink for deserialization. Similar for serialize. This was not
> great, because maps are not ideal and it was a big change as it needed core
> Flink interface changes
>   2.  We then moved the Avro Apicurio format to the Kafka connector and
> looked to discover a new record based de/serialization interface. So we
> could pass down the record (containing the headers) rather than the
> payload. This did not work, because there is a dependence on the Avro
> connector that is not  aware of the new interface.
>   3.  We considered using Thread local storage to pass the headers, we did
> not like this as there was a risk of memory leaks if we did not manage the
> thread well, also the contract is hidden.
>   4.  We then came up with the current design that augments the
> deserialization in the Kafka connector in a new discovered record based
> deserialization, it then takes the headers out in the schema coder, leaving
> the message as it was. Similar for serialization.
>
>
> One piece I need to work out the details of, is how to work when there are
> 2 implementations that can be discovered, probably using an augmented
> format name as a factory identifier,
>
> I hope to put up a new design in the Flip by the end of next week, for
> wider review,
>     Kind regards, David.
>
>
> From: Kevin Lam <ke...@shopify.com.INVALID>
> Date: Monday, 8 July 2024 at 21:16
> To: dev@flink.apache.org <de...@flink.apache.org>
> Subject: [EXTERNAL] Re: [DISCUSS] FLIP-XXX Apicurio-avro format
> Hi David,
>
> Any updates on the Kafka Message Header support? I am also interested in
> supporting headers with the Flink SQL Formats:
> https://lists.apache.org/thread/spl88o63sjm2dv4l5no0ym632d2yt2o6
>
> On Fri, Jun 14, 2024 at 6:10 AM David Radley <da...@uk.ibm.com>
> wrote:
>
> > Hi everyone,
> > I have talked with Chesnay and Danny offline. Danny and I were not very
> > happy with the passing Maps around, and were looking for a neater design.
> > Chesnay suggested that we could move the new format to the Kafka
> connector,
> > then pass the Kafka record down to the deserialize logic so it can make
> use
> > of the headers during deserialization and serialisation.
> >
> > I think this is a neat idea. This would mean:
> > - the Kafka connector code would need to be updated to pass down the
> Kafka
> > record
> > - there would be the Avro Apicurio format and SQL in the kafka
> repository.
> > We feel it is unlikely to want to use the Apicurio registry with files,
> as
> > the Avro format could be used.
> >
> > Unfortunately I have found that this as not so straight forward to
> > implement as the Avro Apicurio format uses the Avro format, which is tied
> > to the DeserializationSchema. We were hoping to have a new decoding
> > implementation that would pass down the Kafka record rather than the
> > payload. This does not appear possible without a Avro format change.
> >
> >
> > Inspired by this idea, I notice that
> > KafkaValueOnlyRecordDeserializerWrapper<T> extends
> > KafkaValueOnlyDeserializerWrapper
> >
> > Does
> >
> > deserializer.deserialize(record.topic(),record.value())
> >
> >
> >
> > I am investigating If I can add a factory/reflection to provide an
> > alternative
> > Implementation that will pass the record based (the kafka record is not
> > serializable so I will pick what we need and deserialize) as a byte
> array.
> >
> > I would need to do this 4 times (value ,key for deserialisation and
> > serialisation. To do this I would need to convert the record into a byte
> > array, so it fits into the existing interface (DeserializationSchema).  I
> > think this could be a way through, to avoid using maps and avoid changing
> > the existing Avro format and avoid change any core Flink interfaces.
> >
> > I am going to prototype this idea. WDYT?
> >
> > My thanks go to Chesnay and Danny for their support and insight around
> > this Flip,
> >    Kind regards, David.
> >
> >
> >
> >
> >
> >
> > From: David Radley <da...@uk.ibm.com>
> > Date: Wednesday, 29 May 2024 at 11:39
> > To: dev@flink.apache.org <de...@flink.apache.org>
> > Subject: [EXTERNAL] RE: [DISCUSS] FLIP-XXX Apicurio-avro format
> > Hi Danny,
> > Thank you for your feedback on this.
> >
> > I agree that using maps has pros and cons. The maps are flexible, but do
> > require the sender and receiver to know what is in the map.
> >
> > When you say “That sounds like it would fit in better, I assume we cannot
> > just take that approach?” The motivation behind this Flip is to support
> the
> > headers which is the usual way that Apicurio runs. We will support the
> > “schema id in the payload” as well.
> >
> > I agree with you when you say “ I am not 100% happy with the solution
> but I
> > cannot offer a better option.” – this is a pragmatic way we have found to
> > solve this issue. I am open to any suggestions to improve this as well.
> >
> > If we are going with the maps design (which is the best we have at the
> > moment) ; it would be good to have the Flink core changes in base Flink
> > version 2.0 as this would mean we do not need to use reflection in a
> Flink
> > Kafka version 2 connector to work out if the runtime Flink has the new
> > methods.
> >
> > At this stage we only have one committer (yourself) backing this. Do you
> > know of other 2 committers who would support this Flip?
> >
> >      Kind regards, David.
> >
> >
> >
> > From: Danny Cranmer <da...@apache.org>
> > Date: Friday, 24 May 2024 at 19:32
> > To: dev@flink.apache.org <de...@flink.apache.org>
> > Subject: [EXTERNAL] Re: [DISCUSS] FLIP-XXX Apicurio-avro format
> > Hello,
> >
> > > I am curious what you mean by abused.
> >
> > I just meant we will end up adding more and more fields to this map over
> > time, and it may be hard to undo.
> >
> > > For Apicurio it can be sent at the start of the payload like Confluent
> > Avro does. Confluent Avro have a magic byte followed by 4 bytes of schema
> > id, at the start of the payload. Apicurio clients and SerDe libraries can
> > be configured to not put the schema id in the headers in which case there
> > is a magic byte followed by an 8 byte schema at the start of the payload.
> > In the deserialization case, we would not need to look at the headers –
> > though the encoding is also in the headers.
> >
> > That sounds like it would fit in better, I assume we cannot just take
> that
> > approach?
> >
> > Thanks for the discussion. I am not 100% happy with the solution but I
> > cannot offer a better option. I would be interested to hear if others
> have
> > any suggestions. Playing devil's advocate against myself, we pass maps
> > around to configure connectors so it is not too far away from that.
> >
> > Thanks,
> > Danny
> >
> >
> > On Fri, May 24, 2024 at 2:23 PM David Radley <da...@uk.ibm.com>
> > wrote:
> >
> > > Hi Danny,
> > > No worries, thanks for replying. I have working prototype code that is
> > > being reviewed. It needs some cleaning up and more complete testing
> > before
> > > it is ready, but will give you the general idea [1][2] to help to
> assess
> > > this approach.
> > >
> > >
> > > I am curious what you mean by abused. I guess the approaches are
> between
> > > generic map, mechanism vs a more particular more granular things being
> > > passed that might be used by another connector.
> > >
> > > Your first question:
> > > “how would this work if the schema ID is not in the Kafka headers, as
> > > hinted to in the FLIP "usually the global ID in a Kafka header"?
> > >
> > > For Apicurio it can be sent at the start of the payload like Confluent
> > > Avro does. Confluent Avro have a magic byte followed by 4 bytes of
> schema
> > > id, at the start of the payload. Apicurio clients and SerDe libraries
> can
> > > be configured to not put the schema id in the headers in which case
> there
> > > is a magic byte followed by an 8 byte schema at the start of the
> payload.
> > > In the deserialization case, we would not need to look at the headers –
> > > though the encoding is also in the headers.
> > >
> > > Your second question:
> > > “I am wondering if there are any other instances where the source would
> > be
> > > aware of the schema ID and pass it through in this way?
> > > ”
> > > The examples I can think of are:
> > > - Avro can send the complete schema in a header, this is not
> recommended
> > > but in theory fits the need for a message payload to require something
> > else
> > > to get the structure.
> > > - I see [2] that Apicurio Protobuf uses headers.
> > > - it might be that other message queuing projects like Rabbit MQ would
> > > need this to be able to support Apicurio Avro & protobuf.
> > >
> > > Kind regards, David,
> > >
> > >
> > >
> > >
> > > [1] https://github.com/apache/flink/pull/24715
> > > [2] https://github.com/apache/flink-connector-kafka/pull/99
> > > [3]
> > >
> >
> https://www.apicur.io/registry/docs/apicurio-registry/2.5.x/getting-started/assembly-configuring-kafka-client-serdes.html#registry-serdes-types-json_registry
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > From: Danny Cranmer <da...@apache.org>
> > > Date: Friday, 24 May 2024 at 12:22
> > > To: dev@flink.apache.org <de...@flink.apache.org>
> > > Subject: [EXTERNAL] Re: [DISCUSS] FLIP-XXX Apicurio-avro format
> > > Hello,
> > >
> > > Apologies, I am on vacation and have limited access to email.
> > >
> > > I can see the logic here and why you ended up where you did. I can also
> > see
> > > there are other useful metadata fields that we might want to pass
> > through,
> > > which might result in this Map being abused (Kafka Topic, Kinesis
> Shard,
> > > etc).
> > >
> > > I have a follow up question, how would this work if the schema ID is
> not
> > in
> > > the Kafka headers, as hinted to in the FLIP "usually the global ID in a
> > > Kafka header"? I am wondering if there are any other instances where
> the
> > > source would be aware of the schema ID and pass it through in this way?
> > >
> > > Thanks,
> > > Danny
> > >
> > >
> > >
> > > On Wed, May 22, 2024 at 3:43 PM David Radley <da...@uk.ibm.com>
> > > wrote:
> > >
> > > > Hi Danny,
> > > > Did you have a chance you have a look at my responses to your
> > feedback? I
> > > > am hoping to keep the momentum going on this one,   kind regards,
> > David.
> > > >
> > > >
> > > > From: David Radley <da...@uk.ibm.com>
> > > > Date: Tuesday, 14 May 2024 at 17:21
> > > > To: dev@flink.apache.org <de...@flink.apache.org>
> > > > Subject: [EXTERNAL] [DISCUSS] FLIP-XXX Apicurio-avro format
> > > > Hi Danny,
> > > >
> > > > Thank you very much for the feedback and your support. I have copied
> > your
> > > > feedback from the VOTE thread to this discussion thread, so we can
> > > continue
> > > > our discussions off the VOTE thread.
> > > >
> > > >
> > > >
> > > > Your feedback:
> > > >
> > > > Thanks for Driving this David. I am +1 for adding support for the new
> > > >
> > > > format, however have some questions/suggestions on the details.
> > > >
> > > >
> > > >
> > > > 1. Passing around Map<String, Object> additionalInputProperties
> feels a
> > > bit
> > > >
> > > > dirty. It looks like this is mainly for the Kafka connector. This
> > > connector
> > > >
> > > > already has a de/serialization schema extension to access record
> > > >
> > > > headers, KafkaRecordDeserializationSchema [1], can we use this
> instead?
> > > >
> > > > 2. Can you elaborate why we need to change the SchemaCoder interface?
> > > Again
> > > >
> > > > I am not a fan of adding these Map parameters
> > > >
> > > > 3. I assume this integration will go into the core Flink repo under
> > > >
> > > > flink-formats [2], and not be a separate repository like the
> > connectors?
> > > >
> > > >
> > > >
> > > > My response:
> > > >
> > > > Addressing 1. and 2.
> > > >
> > > > I agree that sending maps around is a bit dirty. If we can see a
> better
> > > > way that would be great. I was looking for a way to pass this kafka
> > > header
> > > > information in a non-Kafka way - the most obvious way I could think
> was
> > > as
> > > > a map. Here are the main considerations I saw, if I have missed
> > anything
> > > or
> > > > could improve something I would be grateful for any further feedback.
> > > >
> > > >
> > > >
> > > >   *   I see KafkaRecordDeserializationSchema is a Kafka interface
> that
> > > > works at the Kafka record level (so includes the headers). We need a
> > > > mechanism to send over the headers from the Kafka record to Flink
> > > >   *   Flink core is not aware of Kafka headers, and I did not want to
> > add
> > > > a Kafka dependancy to core flink.
> > > >   *   The formats are stateless so it did not appear to be in fitting
> > > with
> > > > the Flink architecture to pass through header information to stash in
> > > state
> > > > in the format waiting for the deserialise to be subsequently called
> to
> > > pick
> > > > up the header information.
> > > >   *   We could have used Thread local storage to stash the header
> > > content,
> > > > but this would be extra state to manage; and this would seem like an
> > > > obtrusive change.
> > > >   *   The SchemaCoder deserialise is where Confluent Avro gets the
> > schema
> > > > id from the payload, so it can lookup the schema. In line with this
> > > > approach it made sense to extend the deserialise so it had the header
> > > > contents so the Apicurio Avro format could lookup the schema.
> > > >   *   I did not want to have Apicurio specific logic in the Kafka
> > > > connector, if we did we could pull out the appropriate headers and
> only
> > > > send over the schema ids.
> > > >   *   For deserialise, the schema id we are interested in is the one
> in
> > > > the Kafka headers on the message and is for the writer schema (an
> Avro
> > > > format concept) currently used by the confluent-avro format in
> > > deserialize.
> > > >   *   For serialise the schema ids need to be obtained from apicurio
> > then
> > > > passed through to Kafka.
> > > >   *   For serialise there is existing logic around handling the
> > metadata
> > > > which includes passing the headers. But the presence of the metadata
> > > would
> > > > imply we have a metadata column. Maybe a change to the metadata
> > mechanism
> > > > may have allowed to use to pass the headers, but not create a
> metadata
> > > > column; instead I pass through the additional headers in a map to be
> > > > appended.
> > > >
> > > >
> > > >
> > > > 3.
> > > >
> > > > Yes this integration will go into the core Flink repo under
> > > >
> > > > flink-formats and sit next to the confluent-avro format. The Avro
> > format
> > > > has the concept of a Registry and drives the confluent-avro format.
> The
> > > > Apicurio Avro format will use the same approach.
> > > >
> > > > Unless otherwise stated above:
> > > >
> > > > IBM United Kingdom Limited
> > > > Registered in England and Wales with number 741598
> > > > Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6
> 3AU
> > > >
> > > > Unless otherwise stated above:
> > > >
> > > > IBM United Kingdom Limited
> > > > Registered in England and Wales with number 741598
> > > > Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6
> 3AU
> > > >
> > >
> > > Unless otherwise stated above:
> > >
> > > IBM United Kingdom Limited
> > > Registered in England and Wales with number 741598
> > > Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU
> > >
> >
> > Unless otherwise stated above:
> >
> > IBM United Kingdom Limited
> > Registered in England and Wales with number 741598
> > Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU
> >
> > Unless otherwise stated above:
> >
> > IBM United Kingdom Limited
> > Registered in England and Wales with number 741598
> > Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU
> >
>
> Unless otherwise stated above:
>
> IBM United Kingdom Limited
> Registered in England and Wales with number 741598
> Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU
> Unless otherwise stated above:
>
> IBM United Kingdom Limited
> Registered in England and Wales with number 741598
> Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU
>

RE: [DISCUSS] FLIP-XXX Apicurio-avro format

Posted by David Radley <da...@uk.ibm.com>.
Hello all,

I have prototyped the new design and confirmed it works. I have documented the design in google doc https://docs.google.com/document/d/1J1E-cE-X2H3-kw4rNjLn71OGPQk_Yl1iGX4-eCHWLgE/edit

By copy Martijn: please could you move this content over to replace the content of https://cwiki.apache.org/confluence/display/FLINK/FLIP-454%3A+New+Apicurio+Avro+format

As the design is significantly different, there will be a new period of discussion before going to vote.

Fyi I will be on vacation after this week until the 13th of August.

Kind regards, David.



From: David Radley <da...@uk.ibm.com>
Date: Tuesday, 9 July 2024 at 11:17
To: dev@flink.apache.org <de...@flink.apache.org>
Subject: [EXTERNAL] RE: [DISCUSS] FLIP-XXX Apicurio-avro format
Hi Kevin,
I have agreed a design with Chesnay and Danny. I am implementing a prototype, to prove it works,  then will update the Flip text with the new design. Initial testing is showing it working.

Here is a quick history so you can understand our current thinking.

  1.  Initially we passed maps for header information from the kafka connector to Flink for deserialization. Similar for serialize. This was not great, because maps are not ideal and it was a big change as it needed core Flink interface changes
  2.  We then moved the Avro Apicurio format to the Kafka connector and looked to discover a new record based de/serialization interface. So we could pass down the record (containing the headers) rather than the payload. This did not work, because there is a dependence on the Avro connector that is not  aware of the new interface.
  3.  We considered using Thread local storage to pass the headers, we did not like this as there was a risk of memory leaks if we did not manage the thread well, also the contract is hidden.
  4.  We then came up with the current design that augments the deserialization in the Kafka connector in a new discovered record based deserialization, it then takes the headers out in the schema coder, leaving the message as it was. Similar for serialization.


One piece I need to work out the details of, is how to work when there are 2 implementations that can be discovered, probably using an augmented format name as a factory identifier,

I hope to put up a new design in the Flip by the end of next week, for wider review,
    Kind regards, David.


From: Kevin Lam <ke...@shopify.com.INVALID>
Date: Monday, 8 July 2024 at 21:16
To: dev@flink.apache.org <de...@flink.apache.org>
Subject: [EXTERNAL] Re: [DISCUSS] FLIP-XXX Apicurio-avro format
Hi David,

Any updates on the Kafka Message Header support? I am also interested in
supporting headers with the Flink SQL Formats:
https://lists.apache.org/thread/spl88o63sjm2dv4l5no0ym632d2yt2o6

On Fri, Jun 14, 2024 at 6:10 AM David Radley <da...@uk.ibm.com>
wrote:

> Hi everyone,
> I have talked with Chesnay and Danny offline. Danny and I were not very
> happy with the passing Maps around, and were looking for a neater design.
> Chesnay suggested that we could move the new format to the Kafka connector,
> then pass the Kafka record down to the deserialize logic so it can make use
> of the headers during deserialization and serialisation.
>
> I think this is a neat idea. This would mean:
> - the Kafka connector code would need to be updated to pass down the Kafka
> record
> - there would be the Avro Apicurio format and SQL in the kafka repository.
> We feel it is unlikely to want to use the Apicurio registry with files, as
> the Avro format could be used.
>
> Unfortunately I have found that this as not so straight forward to
> implement as the Avro Apicurio format uses the Avro format, which is tied
> to the DeserializationSchema. We were hoping to have a new decoding
> implementation that would pass down the Kafka record rather than the
> payload. This does not appear possible without a Avro format change.
>
>
> Inspired by this idea, I notice that
> KafkaValueOnlyRecordDeserializerWrapper<T> extends
> KafkaValueOnlyDeserializerWrapper
>
> Does
>
> deserializer.deserialize(record.topic(),record.value())
>
>
>
> I am investigating If I can add a factory/reflection to provide an
> alternative
> Implementation that will pass the record based (the kafka record is not
> serializable so I will pick what we need and deserialize) as a byte array.
>
> I would need to do this 4 times (value ,key for deserialisation and
> serialisation. To do this I would need to convert the record into a byte
> array, so it fits into the existing interface (DeserializationSchema).  I
> think this could be a way through, to avoid using maps and avoid changing
> the existing Avro format and avoid change any core Flink interfaces.
>
> I am going to prototype this idea. WDYT?
>
> My thanks go to Chesnay and Danny for their support and insight around
> this Flip,
>    Kind regards, David.
>
>
>
>
>
>
> From: David Radley <da...@uk.ibm.com>
> Date: Wednesday, 29 May 2024 at 11:39
> To: dev@flink.apache.org <de...@flink.apache.org>
> Subject: [EXTERNAL] RE: [DISCUSS] FLIP-XXX Apicurio-avro format
> Hi Danny,
> Thank you for your feedback on this.
>
> I agree that using maps has pros and cons. The maps are flexible, but do
> require the sender and receiver to know what is in the map.
>
> When you say “That sounds like it would fit in better, I assume we cannot
> just take that approach?” The motivation behind this Flip is to support the
> headers which is the usual way that Apicurio runs. We will support the
> “schema id in the payload” as well.
>
> I agree with you when you say “ I am not 100% happy with the solution but I
> cannot offer a better option.” – this is a pragmatic way we have found to
> solve this issue. I am open to any suggestions to improve this as well.
>
> If we are going with the maps design (which is the best we have at the
> moment) ; it would be good to have the Flink core changes in base Flink
> version 2.0 as this would mean we do not need to use reflection in a Flink
> Kafka version 2 connector to work out if the runtime Flink has the new
> methods.
>
> At this stage we only have one committer (yourself) backing this. Do you
> know of other 2 committers who would support this Flip?
>
>      Kind regards, David.
>
>
>
> From: Danny Cranmer <da...@apache.org>
> Date: Friday, 24 May 2024 at 19:32
> To: dev@flink.apache.org <de...@flink.apache.org>
> Subject: [EXTERNAL] Re: [DISCUSS] FLIP-XXX Apicurio-avro format
> Hello,
>
> > I am curious what you mean by abused.
>
> I just meant we will end up adding more and more fields to this map over
> time, and it may be hard to undo.
>
> > For Apicurio it can be sent at the start of the payload like Confluent
> Avro does. Confluent Avro have a magic byte followed by 4 bytes of schema
> id, at the start of the payload. Apicurio clients and SerDe libraries can
> be configured to not put the schema id in the headers in which case there
> is a magic byte followed by an 8 byte schema at the start of the payload.
> In the deserialization case, we would not need to look at the headers –
> though the encoding is also in the headers.
>
> That sounds like it would fit in better, I assume we cannot just take that
> approach?
>
> Thanks for the discussion. I am not 100% happy with the solution but I
> cannot offer a better option. I would be interested to hear if others have
> any suggestions. Playing devil's advocate against myself, we pass maps
> around to configure connectors so it is not too far away from that.
>
> Thanks,
> Danny
>
>
> On Fri, May 24, 2024 at 2:23 PM David Radley <da...@uk.ibm.com>
> wrote:
>
> > Hi Danny,
> > No worries, thanks for replying. I have working prototype code that is
> > being reviewed. It needs some cleaning up and more complete testing
> before
> > it is ready, but will give you the general idea [1][2] to help to assess
> > this approach.
> >
> >
> > I am curious what you mean by abused. I guess the approaches are between
> > generic map, mechanism vs a more particular more granular things being
> > passed that might be used by another connector.
> >
> > Your first question:
> > “how would this work if the schema ID is not in the Kafka headers, as
> > hinted to in the FLIP "usually the global ID in a Kafka header"?
> >
> > For Apicurio it can be sent at the start of the payload like Confluent
> > Avro does. Confluent Avro have a magic byte followed by 4 bytes of schema
> > id, at the start of the payload. Apicurio clients and SerDe libraries can
> > be configured to not put the schema id in the headers in which case there
> > is a magic byte followed by an 8 byte schema at the start of the payload.
> > In the deserialization case, we would not need to look at the headers –
> > though the encoding is also in the headers.
> >
> > Your second question:
> > “I am wondering if there are any other instances where the source would
> be
> > aware of the schema ID and pass it through in this way?
> > ”
> > The examples I can think of are:
> > - Avro can send the complete schema in a header, this is not recommended
> > but in theory fits the need for a message payload to require something
> else
> > to get the structure.
> > - I see [2] that Apicurio Protobuf uses headers.
> > - it might be that other message queuing projects like Rabbit MQ would
> > need this to be able to support Apicurio Avro & protobuf.
> >
> > Kind regards, David,
> >
> >
> >
> >
> > [1] https://github.com/apache/flink/pull/24715
> > [2] https://github.com/apache/flink-connector-kafka/pull/99
> > [3]
> >
> https://www.apicur.io/registry/docs/apicurio-registry/2.5.x/getting-started/assembly-configuring-kafka-client-serdes.html#registry-serdes-types-json_registry
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > From: Danny Cranmer <da...@apache.org>
> > Date: Friday, 24 May 2024 at 12:22
> > To: dev@flink.apache.org <de...@flink.apache.org>
> > Subject: [EXTERNAL] Re: [DISCUSS] FLIP-XXX Apicurio-avro format
> > Hello,
> >
> > Apologies, I am on vacation and have limited access to email.
> >
> > I can see the logic here and why you ended up where you did. I can also
> see
> > there are other useful metadata fields that we might want to pass
> through,
> > which might result in this Map being abused (Kafka Topic, Kinesis Shard,
> > etc).
> >
> > I have a follow up question, how would this work if the schema ID is not
> in
> > the Kafka headers, as hinted to in the FLIP "usually the global ID in a
> > Kafka header"? I am wondering if there are any other instances where the
> > source would be aware of the schema ID and pass it through in this way?
> >
> > Thanks,
> > Danny
> >
> >
> >
> > On Wed, May 22, 2024 at 3:43 PM David Radley <da...@uk.ibm.com>
> > wrote:
> >
> > > Hi Danny,
> > > Did you have a chance you have a look at my responses to your
> feedback? I
> > > am hoping to keep the momentum going on this one,   kind regards,
> David.
> > >
> > >
> > > From: David Radley <da...@uk.ibm.com>
> > > Date: Tuesday, 14 May 2024 at 17:21
> > > To: dev@flink.apache.org <de...@flink.apache.org>
> > > Subject: [EXTERNAL] [DISCUSS] FLIP-XXX Apicurio-avro format
> > > Hi Danny,
> > >
> > > Thank you very much for the feedback and your support. I have copied
> your
> > > feedback from the VOTE thread to this discussion thread, so we can
> > continue
> > > our discussions off the VOTE thread.
> > >
> > >
> > >
> > > Your feedback:
> > >
> > > Thanks for Driving this David. I am +1 for adding support for the new
> > >
> > > format, however have some questions/suggestions on the details.
> > >
> > >
> > >
> > > 1. Passing around Map<String, Object> additionalInputProperties feels a
> > bit
> > >
> > > dirty. It looks like this is mainly for the Kafka connector. This
> > connector
> > >
> > > already has a de/serialization schema extension to access record
> > >
> > > headers, KafkaRecordDeserializationSchema [1], can we use this instead?
> > >
> > > 2. Can you elaborate why we need to change the SchemaCoder interface?
> > Again
> > >
> > > I am not a fan of adding these Map parameters
> > >
> > > 3. I assume this integration will go into the core Flink repo under
> > >
> > > flink-formats [2], and not be a separate repository like the
> connectors?
> > >
> > >
> > >
> > > My response:
> > >
> > > Addressing 1. and 2.
> > >
> > > I agree that sending maps around is a bit dirty. If we can see a better
> > > way that would be great. I was looking for a way to pass this kafka
> > header
> > > information in a non-Kafka way - the most obvious way I could think was
> > as
> > > a map. Here are the main considerations I saw, if I have missed
> anything
> > or
> > > could improve something I would be grateful for any further feedback.
> > >
> > >
> > >
> > >   *   I see KafkaRecordDeserializationSchema is a Kafka interface that
> > > works at the Kafka record level (so includes the headers). We need a
> > > mechanism to send over the headers from the Kafka record to Flink
> > >   *   Flink core is not aware of Kafka headers, and I did not want to
> add
> > > a Kafka dependancy to core flink.
> > >   *   The formats are stateless so it did not appear to be in fitting
> > with
> > > the Flink architecture to pass through header information to stash in
> > state
> > > in the format waiting for the deserialise to be subsequently called to
> > pick
> > > up the header information.
> > >   *   We could have used Thread local storage to stash the header
> > content,
> > > but this would be extra state to manage; and this would seem like an
> > > obtrusive change.
> > >   *   The SchemaCoder deserialise is where Confluent Avro gets the
> schema
> > > id from the payload, so it can lookup the schema. In line with this
> > > approach it made sense to extend the deserialise so it had the header
> > > contents so the Apicurio Avro format could lookup the schema.
> > >   *   I did not want to have Apicurio specific logic in the Kafka
> > > connector, if we did we could pull out the appropriate headers and only
> > > send over the schema ids.
> > >   *   For deserialise, the schema id we are interested in is the one in
> > > the Kafka headers on the message and is for the writer schema (an Avro
> > > format concept) currently used by the confluent-avro format in
> > deserialize.
> > >   *   For serialise the schema ids need to be obtained from apicurio
> then
> > > passed through to Kafka.
> > >   *   For serialise there is existing logic around handling the
> metadata
> > > which includes passing the headers. But the presence of the metadata
> > would
> > > imply we have a metadata column. Maybe a change to the metadata
> mechanism
> > > may have allowed to use to pass the headers, but not create a metadata
> > > column; instead I pass through the additional headers in a map to be
> > > appended.
> > >
> > >
> > >
> > > 3.
> > >
> > > Yes this integration will go into the core Flink repo under
> > >
> > > flink-formats and sit next to the confluent-avro format. The Avro
> format
> > > has the concept of a Registry and drives the confluent-avro format. The
> > > Apicurio Avro format will use the same approach.
> > >
> > > Unless otherwise stated above:
> > >
> > > IBM United Kingdom Limited
> > > Registered in England and Wales with number 741598
> > > Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU
> > >
> > > Unless otherwise stated above:
> > >
> > > IBM United Kingdom Limited
> > > Registered in England and Wales with number 741598
> > > Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU
> > >
> >
> > Unless otherwise stated above:
> >
> > IBM United Kingdom Limited
> > Registered in England and Wales with number 741598
> > Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU
> >
>
> Unless otherwise stated above:
>
> IBM United Kingdom Limited
> Registered in England and Wales with number 741598
> Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU
>
> Unless otherwise stated above:
>
> IBM United Kingdom Limited
> Registered in England and Wales with number 741598
> Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU
>

Unless otherwise stated above:

IBM United Kingdom Limited
Registered in England and Wales with number 741598
Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU

Unless otherwise stated above:

IBM United Kingdom Limited
Registered in England and Wales with number 741598
Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU

RE: [DISCUSS] FLIP-XXX Apicurio-avro format

Posted by David Radley <da...@uk.ibm.com>.
Hi Kevin,
I have agreed a design with Chesnay and Danny. I am implementing a prototype, to prove it works,  then will update the Flip text with the new design. Initial testing is showing it working.

Here is a quick history so you can understand our current thinking.

  1.  Initially we passed maps for header information from the kafka connector to Flink for deserialization. Similar for serialize. This was not great, because maps are not ideal and it was a big change as it needed core Flink interface changes
  2.  We then moved the Avro Apicurio format to the Kafka connector and looked to discover a new record based de/serialization interface. So we could pass down the record (containing the headers) rather than the payload. This did not work, because there is a dependence on the Avro connector that is not  aware of the new interface.
  3.  We considered using Thread local storage to pass the headers, we did not like this as there was a risk of memory leaks if we did not manage the thread well, also the contract is hidden.
  4.  We then came up with the current design that augments the deserialization in the Kafka connector in a new discovered record based deserialization, it then takes the headers out in the schema coder, leaving the message as it was. Similar for serialization.


One piece I need to work out the details of, is how to work when there are 2 implementations that can be discovered, probably using an augmented format name as a factory identifier,

I hope to put up a new design in the Flip by the end of next week, for wider review,
    Kind regards, David.


From: Kevin Lam <ke...@shopify.com.INVALID>
Date: Monday, 8 July 2024 at 21:16
To: dev@flink.apache.org <de...@flink.apache.org>
Subject: [EXTERNAL] Re: [DISCUSS] FLIP-XXX Apicurio-avro format
Hi David,

Any updates on the Kafka Message Header support? I am also interested in
supporting headers with the Flink SQL Formats:
https://lists.apache.org/thread/spl88o63sjm2dv4l5no0ym632d2yt2o6

On Fri, Jun 14, 2024 at 6:10 AM David Radley <da...@uk.ibm.com>
wrote:

> Hi everyone,
> I have talked with Chesnay and Danny offline. Danny and I were not very
> happy with the passing Maps around, and were looking for a neater design.
> Chesnay suggested that we could move the new format to the Kafka connector,
> then pass the Kafka record down to the deserialize logic so it can make use
> of the headers during deserialization and serialisation.
>
> I think this is a neat idea. This would mean:
> - the Kafka connector code would need to be updated to pass down the Kafka
> record
> - there would be the Avro Apicurio format and SQL in the kafka repository.
> We feel it is unlikely to want to use the Apicurio registry with files, as
> the Avro format could be used.
>
> Unfortunately I have found that this as not so straight forward to
> implement as the Avro Apicurio format uses the Avro format, which is tied
> to the DeserializationSchema. We were hoping to have a new decoding
> implementation that would pass down the Kafka record rather than the
> payload. This does not appear possible without a Avro format change.
>
>
> Inspired by this idea, I notice that
> KafkaValueOnlyRecordDeserializerWrapper<T> extends
> KafkaValueOnlyDeserializerWrapper
>
> Does
>
> deserializer.deserialize(record.topic(),record.value())
>
>
>
> I am investigating If I can add a factory/reflection to provide an
> alternative
> Implementation that will pass the record based (the kafka record is not
> serializable so I will pick what we need and deserialize) as a byte array.
>
> I would need to do this 4 times (value ,key for deserialisation and
> serialisation. To do this I would need to convert the record into a byte
> array, so it fits into the existing interface (DeserializationSchema).  I
> think this could be a way through, to avoid using maps and avoid changing
> the existing Avro format and avoid change any core Flink interfaces.
>
> I am going to prototype this idea. WDYT?
>
> My thanks go to Chesnay and Danny for their support and insight around
> this Flip,
>    Kind regards, David.
>
>
>
>
>
>
> From: David Radley <da...@uk.ibm.com>
> Date: Wednesday, 29 May 2024 at 11:39
> To: dev@flink.apache.org <de...@flink.apache.org>
> Subject: [EXTERNAL] RE: [DISCUSS] FLIP-XXX Apicurio-avro format
> Hi Danny,
> Thank you for your feedback on this.
>
> I agree that using maps has pros and cons. The maps are flexible, but do
> require the sender and receiver to know what is in the map.
>
> When you say “That sounds like it would fit in better, I assume we cannot
> just take that approach?” The motivation behind this Flip is to support the
> headers which is the usual way that Apicurio runs. We will support the
> “schema id in the payload” as well.
>
> I agree with you when you say “ I am not 100% happy with the solution but I
> cannot offer a better option.” – this is a pragmatic way we have found to
> solve this issue. I am open to any suggestions to improve this as well.
>
> If we are going with the maps design (which is the best we have at the
> moment) ; it would be good to have the Flink core changes in base Flink
> version 2.0 as this would mean we do not need to use reflection in a Flink
> Kafka version 2 connector to work out if the runtime Flink has the new
> methods.
>
> At this stage we only have one committer (yourself) backing this. Do you
> know of other 2 committers who would support this Flip?
>
>      Kind regards, David.
>
>
>
> From: Danny Cranmer <da...@apache.org>
> Date: Friday, 24 May 2024 at 19:32
> To: dev@flink.apache.org <de...@flink.apache.org>
> Subject: [EXTERNAL] Re: [DISCUSS] FLIP-XXX Apicurio-avro format
> Hello,
>
> > I am curious what you mean by abused.
>
> I just meant we will end up adding more and more fields to this map over
> time, and it may be hard to undo.
>
> > For Apicurio it can be sent at the start of the payload like Confluent
> Avro does. Confluent Avro have a magic byte followed by 4 bytes of schema
> id, at the start of the payload. Apicurio clients and SerDe libraries can
> be configured to not put the schema id in the headers in which case there
> is a magic byte followed by an 8 byte schema at the start of the payload.
> In the deserialization case, we would not need to look at the headers –
> though the encoding is also in the headers.
>
> That sounds like it would fit in better, I assume we cannot just take that
> approach?
>
> Thanks for the discussion. I am not 100% happy with the solution but I
> cannot offer a better option. I would be interested to hear if others have
> any suggestions. Playing devil's advocate against myself, we pass maps
> around to configure connectors so it is not too far away from that.
>
> Thanks,
> Danny
>
>
> On Fri, May 24, 2024 at 2:23 PM David Radley <da...@uk.ibm.com>
> wrote:
>
> > Hi Danny,
> > No worries, thanks for replying. I have working prototype code that is
> > being reviewed. It needs some cleaning up and more complete testing
> before
> > it is ready, but will give you the general idea [1][2] to help to assess
> > this approach.
> >
> >
> > I am curious what you mean by abused. I guess the approaches are between
> > generic map, mechanism vs a more particular more granular things being
> > passed that might be used by another connector.
> >
> > Your first question:
> > “how would this work if the schema ID is not in the Kafka headers, as
> > hinted to in the FLIP "usually the global ID in a Kafka header"?
> >
> > For Apicurio it can be sent at the start of the payload like Confluent
> > Avro does. Confluent Avro have a magic byte followed by 4 bytes of schema
> > id, at the start of the payload. Apicurio clients and SerDe libraries can
> > be configured to not put the schema id in the headers in which case there
> > is a magic byte followed by an 8 byte schema at the start of the payload.
> > In the deserialization case, we would not need to look at the headers –
> > though the encoding is also in the headers.
> >
> > Your second question:
> > “I am wondering if there are any other instances where the source would
> be
> > aware of the schema ID and pass it through in this way?
> > ”
> > The examples I can think of are:
> > - Avro can send the complete schema in a header, this is not recommended
> > but in theory fits the need for a message payload to require something
> else
> > to get the structure.
> > - I see [2] that Apicurio Protobuf uses headers.
> > - it might be that other message queuing projects like Rabbit MQ would
> > need this to be able to support Apicurio Avro & protobuf.
> >
> > Kind regards, David,
> >
> >
> >
> >
> > [1] https://github.com/apache/flink/pull/24715
> > [2] https://github.com/apache/flink-connector-kafka/pull/99
> > [3]
> >
> https://www.apicur.io/registry/docs/apicurio-registry/2.5.x/getting-started/assembly-configuring-kafka-client-serdes.html#registry-serdes-types-json_registry
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > From: Danny Cranmer <da...@apache.org>
> > Date: Friday, 24 May 2024 at 12:22
> > To: dev@flink.apache.org <de...@flink.apache.org>
> > Subject: [EXTERNAL] Re: [DISCUSS] FLIP-XXX Apicurio-avro format
> > Hello,
> >
> > Apologies, I am on vacation and have limited access to email.
> >
> > I can see the logic here and why you ended up where you did. I can also
> see
> > there are other useful metadata fields that we might want to pass
> through,
> > which might result in this Map being abused (Kafka Topic, Kinesis Shard,
> > etc).
> >
> > I have a follow up question, how would this work if the schema ID is not
> in
> > the Kafka headers, as hinted to in the FLIP "usually the global ID in a
> > Kafka header"? I am wondering if there are any other instances where the
> > source would be aware of the schema ID and pass it through in this way?
> >
> > Thanks,
> > Danny
> >
> >
> >
> > On Wed, May 22, 2024 at 3:43 PM David Radley <da...@uk.ibm.com>
> > wrote:
> >
> > > Hi Danny,
> > > Did you have a chance you have a look at my responses to your
> feedback? I
> > > am hoping to keep the momentum going on this one,   kind regards,
> David.
> > >
> > >
> > > From: David Radley <da...@uk.ibm.com>
> > > Date: Tuesday, 14 May 2024 at 17:21
> > > To: dev@flink.apache.org <de...@flink.apache.org>
> > > Subject: [EXTERNAL] [DISCUSS] FLIP-XXX Apicurio-avro format
> > > Hi Danny,
> > >
> > > Thank you very much for the feedback and your support. I have copied
> your
> > > feedback from the VOTE thread to this discussion thread, so we can
> > continue
> > > our discussions off the VOTE thread.
> > >
> > >
> > >
> > > Your feedback:
> > >
> > > Thanks for Driving this David. I am +1 for adding support for the new
> > >
> > > format, however have some questions/suggestions on the details.
> > >
> > >
> > >
> > > 1. Passing around Map<String, Object> additionalInputProperties feels a
> > bit
> > >
> > > dirty. It looks like this is mainly for the Kafka connector. This
> > connector
> > >
> > > already has a de/serialization schema extension to access record
> > >
> > > headers, KafkaRecordDeserializationSchema [1], can we use this instead?
> > >
> > > 2. Can you elaborate why we need to change the SchemaCoder interface?
> > Again
> > >
> > > I am not a fan of adding these Map parameters
> > >
> > > 3. I assume this integration will go into the core Flink repo under
> > >
> > > flink-formats [2], and not be a separate repository like the
> connectors?
> > >
> > >
> > >
> > > My response:
> > >
> > > Addressing 1. and 2.
> > >
> > > I agree that sending maps around is a bit dirty. If we can see a better
> > > way that would be great. I was looking for a way to pass this kafka
> > header
> > > information in a non-Kafka way - the most obvious way I could think was
> > as
> > > a map. Here are the main considerations I saw, if I have missed
> anything
> > or
> > > could improve something I would be grateful for any further feedback.
> > >
> > >
> > >
> > >   *   I see KafkaRecordDeserializationSchema is a Kafka interface that
> > > works at the Kafka record level (so includes the headers). We need a
> > > mechanism to send over the headers from the Kafka record to Flink
> > >   *   Flink core is not aware of Kafka headers, and I did not want to
> add
> > > a Kafka dependancy to core flink.
> > >   *   The formats are stateless so it did not appear to be in fitting
> > with
> > > the Flink architecture to pass through header information to stash in
> > state
> > > in the format waiting for the deserialise to be subsequently called to
> > pick
> > > up the header information.
> > >   *   We could have used Thread local storage to stash the header
> > content,
> > > but this would be extra state to manage; and this would seem like an
> > > obtrusive change.
> > >   *   The SchemaCoder deserialise is where Confluent Avro gets the
> schema
> > > id from the payload, so it can lookup the schema. In line with this
> > > approach it made sense to extend the deserialise so it had the header
> > > contents so the Apicurio Avro format could lookup the schema.
> > >   *   I did not want to have Apicurio specific logic in the Kafka
> > > connector, if we did we could pull out the appropriate headers and only
> > > send over the schema ids.
> > >   *   For deserialise, the schema id we are interested in is the one in
> > > the Kafka headers on the message and is for the writer schema (an Avro
> > > format concept) currently used by the confluent-avro format in
> > deserialize.
> > >   *   For serialise the schema ids need to be obtained from apicurio
> then
> > > passed through to Kafka.
> > >   *   For serialise there is existing logic around handling the
> metadata
> > > which includes passing the headers. But the presence of the metadata
> > would
> > > imply we have a metadata column. Maybe a change to the metadata
> mechanism
> > > may have allowed to use to pass the headers, but not create a metadata
> > > column; instead I pass through the additional headers in a map to be
> > > appended.
> > >
> > >
> > >
> > > 3.
> > >
> > > Yes this integration will go into the core Flink repo under
> > >
> > > flink-formats and sit next to the confluent-avro format. The Avro
> format
> > > has the concept of a Registry and drives the confluent-avro format. The
> > > Apicurio Avro format will use the same approach.
> > >
> > > Unless otherwise stated above:
> > >
> > > IBM United Kingdom Limited
> > > Registered in England and Wales with number 741598
> > > Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU
> > >
> > > Unless otherwise stated above:
> > >
> > > IBM United Kingdom Limited
> > > Registered in England and Wales with number 741598
> > > Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU
> > >
> >
> > Unless otherwise stated above:
> >
> > IBM United Kingdom Limited
> > Registered in England and Wales with number 741598
> > Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU
> >
>
> Unless otherwise stated above:
>
> IBM United Kingdom Limited
> Registered in England and Wales with number 741598
> Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU
>
> Unless otherwise stated above:
>
> IBM United Kingdom Limited
> Registered in England and Wales with number 741598
> Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU
>

Unless otherwise stated above:

IBM United Kingdom Limited
Registered in England and Wales with number 741598
Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU

Re: [DISCUSS] FLIP-XXX Apicurio-avro format

Posted by Kevin Lam <ke...@shopify.com.INVALID>.
Hi David,

Any updates on the Kafka Message Header support? I am also interested in
supporting headers with the Flink SQL Formats:
https://lists.apache.org/thread/spl88o63sjm2dv4l5no0ym632d2yt2o6

On Fri, Jun 14, 2024 at 6:10 AM David Radley <da...@uk.ibm.com>
wrote:

> Hi everyone,
> I have talked with Chesnay and Danny offline. Danny and I were not very
> happy with the passing Maps around, and were looking for a neater design.
> Chesnay suggested that we could move the new format to the Kafka connector,
> then pass the Kafka record down to the deserialize logic so it can make use
> of the headers during deserialization and serialisation.
>
> I think this is a neat idea. This would mean:
> - the Kafka connector code would need to be updated to pass down the Kafka
> record
> - there would be the Avro Apicurio format and SQL in the kafka repository.
> We feel it is unlikely to want to use the Apicurio registry with files, as
> the Avro format could be used.
>
> Unfortunately I have found that this as not so straight forward to
> implement as the Avro Apicurio format uses the Avro format, which is tied
> to the DeserializationSchema. We were hoping to have a new decoding
> implementation that would pass down the Kafka record rather than the
> payload. This does not appear possible without a Avro format change.
>
>
> Inspired by this idea, I notice that
> KafkaValueOnlyRecordDeserializerWrapper<T> extends
> KafkaValueOnlyDeserializerWrapper
>
> Does
>
> deserializer.deserialize(record.topic(),record.value())
>
>
>
> I am investigating If I can add a factory/reflection to provide an
> alternative
> Implementation that will pass the record based (the kafka record is not
> serializable so I will pick what we need and deserialize) as a byte array.
>
> I would need to do this 4 times (value ,key for deserialisation and
> serialisation. To do this I would need to convert the record into a byte
> array, so it fits into the existing interface (DeserializationSchema).  I
> think this could be a way through, to avoid using maps and avoid changing
> the existing Avro format and avoid change any core Flink interfaces.
>
> I am going to prototype this idea. WDYT?
>
> My thanks go to Chesnay and Danny for their support and insight around
> this Flip,
>    Kind regards, David.
>
>
>
>
>
>
> From: David Radley <da...@uk.ibm.com>
> Date: Wednesday, 29 May 2024 at 11:39
> To: dev@flink.apache.org <de...@flink.apache.org>
> Subject: [EXTERNAL] RE: [DISCUSS] FLIP-XXX Apicurio-avro format
> Hi Danny,
> Thank you for your feedback on this.
>
> I agree that using maps has pros and cons. The maps are flexible, but do
> require the sender and receiver to know what is in the map.
>
> When you say “That sounds like it would fit in better, I assume we cannot
> just take that approach?” The motivation behind this Flip is to support the
> headers which is the usual way that Apicurio runs. We will support the
> “schema id in the payload” as well.
>
> I agree with you when you say “ I am not 100% happy with the solution but I
> cannot offer a better option.” – this is a pragmatic way we have found to
> solve this issue. I am open to any suggestions to improve this as well.
>
> If we are going with the maps design (which is the best we have at the
> moment) ; it would be good to have the Flink core changes in base Flink
> version 2.0 as this would mean we do not need to use reflection in a Flink
> Kafka version 2 connector to work out if the runtime Flink has the new
> methods.
>
> At this stage we only have one committer (yourself) backing this. Do you
> know of other 2 committers who would support this Flip?
>
>      Kind regards, David.
>
>
>
> From: Danny Cranmer <da...@apache.org>
> Date: Friday, 24 May 2024 at 19:32
> To: dev@flink.apache.org <de...@flink.apache.org>
> Subject: [EXTERNAL] Re: [DISCUSS] FLIP-XXX Apicurio-avro format
> Hello,
>
> > I am curious what you mean by abused.
>
> I just meant we will end up adding more and more fields to this map over
> time, and it may be hard to undo.
>
> > For Apicurio it can be sent at the start of the payload like Confluent
> Avro does. Confluent Avro have a magic byte followed by 4 bytes of schema
> id, at the start of the payload. Apicurio clients and SerDe libraries can
> be configured to not put the schema id in the headers in which case there
> is a magic byte followed by an 8 byte schema at the start of the payload.
> In the deserialization case, we would not need to look at the headers –
> though the encoding is also in the headers.
>
> That sounds like it would fit in better, I assume we cannot just take that
> approach?
>
> Thanks for the discussion. I am not 100% happy with the solution but I
> cannot offer a better option. I would be interested to hear if others have
> any suggestions. Playing devil's advocate against myself, we pass maps
> around to configure connectors so it is not too far away from that.
>
> Thanks,
> Danny
>
>
> On Fri, May 24, 2024 at 2:23 PM David Radley <da...@uk.ibm.com>
> wrote:
>
> > Hi Danny,
> > No worries, thanks for replying. I have working prototype code that is
> > being reviewed. It needs some cleaning up and more complete testing
> before
> > it is ready, but will give you the general idea [1][2] to help to assess
> > this approach.
> >
> >
> > I am curious what you mean by abused. I guess the approaches are between
> > generic map, mechanism vs a more particular more granular things being
> > passed that might be used by another connector.
> >
> > Your first question:
> > “how would this work if the schema ID is not in the Kafka headers, as
> > hinted to in the FLIP "usually the global ID in a Kafka header"?
> >
> > For Apicurio it can be sent at the start of the payload like Confluent
> > Avro does. Confluent Avro have a magic byte followed by 4 bytes of schema
> > id, at the start of the payload. Apicurio clients and SerDe libraries can
> > be configured to not put the schema id in the headers in which case there
> > is a magic byte followed by an 8 byte schema at the start of the payload.
> > In the deserialization case, we would not need to look at the headers –
> > though the encoding is also in the headers.
> >
> > Your second question:
> > “I am wondering if there are any other instances where the source would
> be
> > aware of the schema ID and pass it through in this way?
> > ”
> > The examples I can think of are:
> > - Avro can send the complete schema in a header, this is not recommended
> > but in theory fits the need for a message payload to require something
> else
> > to get the structure.
> > - I see [2] that Apicurio Protobuf uses headers.
> > - it might be that other message queuing projects like Rabbit MQ would
> > need this to be able to support Apicurio Avro & protobuf.
> >
> > Kind regards, David,
> >
> >
> >
> >
> > [1] https://github.com/apache/flink/pull/24715
> > [2] https://github.com/apache/flink-connector-kafka/pull/99
> > [3]
> >
> https://www.apicur.io/registry/docs/apicurio-registry/2.5.x/getting-started/assembly-configuring-kafka-client-serdes.html#registry-serdes-types-json_registry
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > From: Danny Cranmer <da...@apache.org>
> > Date: Friday, 24 May 2024 at 12:22
> > To: dev@flink.apache.org <de...@flink.apache.org>
> > Subject: [EXTERNAL] Re: [DISCUSS] FLIP-XXX Apicurio-avro format
> > Hello,
> >
> > Apologies, I am on vacation and have limited access to email.
> >
> > I can see the logic here and why you ended up where you did. I can also
> see
> > there are other useful metadata fields that we might want to pass
> through,
> > which might result in this Map being abused (Kafka Topic, Kinesis Shard,
> > etc).
> >
> > I have a follow up question, how would this work if the schema ID is not
> in
> > the Kafka headers, as hinted to in the FLIP "usually the global ID in a
> > Kafka header"? I am wondering if there are any other instances where the
> > source would be aware of the schema ID and pass it through in this way?
> >
> > Thanks,
> > Danny
> >
> >
> >
> > On Wed, May 22, 2024 at 3:43 PM David Radley <da...@uk.ibm.com>
> > wrote:
> >
> > > Hi Danny,
> > > Did you have a chance you have a look at my responses to your
> feedback? I
> > > am hoping to keep the momentum going on this one,   kind regards,
> David.
> > >
> > >
> > > From: David Radley <da...@uk.ibm.com>
> > > Date: Tuesday, 14 May 2024 at 17:21
> > > To: dev@flink.apache.org <de...@flink.apache.org>
> > > Subject: [EXTERNAL] [DISCUSS] FLIP-XXX Apicurio-avro format
> > > Hi Danny,
> > >
> > > Thank you very much for the feedback and your support. I have copied
> your
> > > feedback from the VOTE thread to this discussion thread, so we can
> > continue
> > > our discussions off the VOTE thread.
> > >
> > >
> > >
> > > Your feedback:
> > >
> > > Thanks for Driving this David. I am +1 for adding support for the new
> > >
> > > format, however have some questions/suggestions on the details.
> > >
> > >
> > >
> > > 1. Passing around Map<String, Object> additionalInputProperties feels a
> > bit
> > >
> > > dirty. It looks like this is mainly for the Kafka connector. This
> > connector
> > >
> > > already has a de/serialization schema extension to access record
> > >
> > > headers, KafkaRecordDeserializationSchema [1], can we use this instead?
> > >
> > > 2. Can you elaborate why we need to change the SchemaCoder interface?
> > Again
> > >
> > > I am not a fan of adding these Map parameters
> > >
> > > 3. I assume this integration will go into the core Flink repo under
> > >
> > > flink-formats [2], and not be a separate repository like the
> connectors?
> > >
> > >
> > >
> > > My response:
> > >
> > > Addressing 1. and 2.
> > >
> > > I agree that sending maps around is a bit dirty. If we can see a better
> > > way that would be great. I was looking for a way to pass this kafka
> > header
> > > information in a non-Kafka way - the most obvious way I could think was
> > as
> > > a map. Here are the main considerations I saw, if I have missed
> anything
> > or
> > > could improve something I would be grateful for any further feedback.
> > >
> > >
> > >
> > >   *   I see KafkaRecordDeserializationSchema is a Kafka interface that
> > > works at the Kafka record level (so includes the headers). We need a
> > > mechanism to send over the headers from the Kafka record to Flink
> > >   *   Flink core is not aware of Kafka headers, and I did not want to
> add
> > > a Kafka dependancy to core flink.
> > >   *   The formats are stateless so it did not appear to be in fitting
> > with
> > > the Flink architecture to pass through header information to stash in
> > state
> > > in the format waiting for the deserialise to be subsequently called to
> > pick
> > > up the header information.
> > >   *   We could have used Thread local storage to stash the header
> > content,
> > > but this would be extra state to manage; and this would seem like an
> > > obtrusive change.
> > >   *   The SchemaCoder deserialise is where Confluent Avro gets the
> schema
> > > id from the payload, so it can lookup the schema. In line with this
> > > approach it made sense to extend the deserialise so it had the header
> > > contents so the Apicurio Avro format could lookup the schema.
> > >   *   I did not want to have Apicurio specific logic in the Kafka
> > > connector, if we did we could pull out the appropriate headers and only
> > > send over the schema ids.
> > >   *   For deserialise, the schema id we are interested in is the one in
> > > the Kafka headers on the message and is for the writer schema (an Avro
> > > format concept) currently used by the confluent-avro format in
> > deserialize.
> > >   *   For serialise the schema ids need to be obtained from apicurio
> then
> > > passed through to Kafka.
> > >   *   For serialise there is existing logic around handling the
> metadata
> > > which includes passing the headers. But the presence of the metadata
> > would
> > > imply we have a metadata column. Maybe a change to the metadata
> mechanism
> > > may have allowed to use to pass the headers, but not create a metadata
> > > column; instead I pass through the additional headers in a map to be
> > > appended.
> > >
> > >
> > >
> > > 3.
> > >
> > > Yes this integration will go into the core Flink repo under
> > >
> > > flink-formats and sit next to the confluent-avro format. The Avro
> format
> > > has the concept of a Registry and drives the confluent-avro format. The
> > > Apicurio Avro format will use the same approach.
> > >
> > > Unless otherwise stated above:
> > >
> > > IBM United Kingdom Limited
> > > Registered in England and Wales with number 741598
> > > Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU
> > >
> > > Unless otherwise stated above:
> > >
> > > IBM United Kingdom Limited
> > > Registered in England and Wales with number 741598
> > > Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU
> > >
> >
> > Unless otherwise stated above:
> >
> > IBM United Kingdom Limited
> > Registered in England and Wales with number 741598
> > Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU
> >
>
> Unless otherwise stated above:
>
> IBM United Kingdom Limited
> Registered in England and Wales with number 741598
> Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU
>
> Unless otherwise stated above:
>
> IBM United Kingdom Limited
> Registered in England and Wales with number 741598
> Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU
>

RE: [DISCUSS] FLIP-XXX Apicurio-avro format

Posted by David Radley <da...@uk.ibm.com>.
Hi everyone,
I have talked with Chesnay and Danny offline. Danny and I were not very happy with the passing Maps around, and were looking for a neater design. Chesnay suggested that we could move the new format to the Kafka connector, then pass the Kafka record down to the deserialize logic so it can make use of the headers during deserialization and serialisation.

I think this is a neat idea. This would mean:
- the Kafka connector code would need to be updated to pass down the Kafka record
- there would be the Avro Apicurio format and SQL in the kafka repository. We feel it is unlikely to want to use the Apicurio registry with files, as the Avro format could be used.

Unfortunately I have found that this as not so straight forward to implement as the Avro Apicurio format uses the Avro format, which is tied to the DeserializationSchema. We were hoping to have a new decoding implementation that would pass down the Kafka record rather than the payload. This does not appear possible without a Avro format change.


Inspired by this idea, I notice that KafkaValueOnlyRecordDeserializerWrapper<T> extends KafkaValueOnlyDeserializerWrapper

Does

deserializer.deserialize(record.topic(),record.value())



I am investigating If I can add a factory/reflection to provide an alternative
Implementation that will pass the record based (the kafka record is not serializable so I will pick what we need and deserialize) as a byte array.

I would need to do this 4 times (value ,key for deserialisation and serialisation. To do this I would need to convert the record into a byte array, so it fits into the existing interface (DeserializationSchema).  I think this could be a way through, to avoid using maps and avoid changing the existing Avro format and avoid change any core Flink interfaces.

I am going to prototype this idea. WDYT?

My thanks go to Chesnay and Danny for their support and insight around this Flip,
   Kind regards, David.






From: David Radley <da...@uk.ibm.com>
Date: Wednesday, 29 May 2024 at 11:39
To: dev@flink.apache.org <de...@flink.apache.org>
Subject: [EXTERNAL] RE: [DISCUSS] FLIP-XXX Apicurio-avro format
Hi Danny,
Thank you for your feedback on this.

I agree that using maps has pros and cons. The maps are flexible, but do require the sender and receiver to know what is in the map.

When you say “That sounds like it would fit in better, I assume we cannot just take that approach?” The motivation behind this Flip is to support the headers which is the usual way that Apicurio runs. We will support the “schema id in the payload” as well.

I agree with you when you say “ I am not 100% happy with the solution but I
cannot offer a better option.” – this is a pragmatic way we have found to solve this issue. I am open to any suggestions to improve this as well.

If we are going with the maps design (which is the best we have at the moment) ; it would be good to have the Flink core changes in base Flink version 2.0 as this would mean we do not need to use reflection in a Flink Kafka version 2 connector to work out if the runtime Flink has the new methods.

At this stage we only have one committer (yourself) backing this. Do you know of other 2 committers who would support this Flip?

     Kind regards, David.



From: Danny Cranmer <da...@apache.org>
Date: Friday, 24 May 2024 at 19:32
To: dev@flink.apache.org <de...@flink.apache.org>
Subject: [EXTERNAL] Re: [DISCUSS] FLIP-XXX Apicurio-avro format
Hello,

> I am curious what you mean by abused.

I just meant we will end up adding more and more fields to this map over
time, and it may be hard to undo.

> For Apicurio it can be sent at the start of the payload like Confluent
Avro does. Confluent Avro have a magic byte followed by 4 bytes of schema
id, at the start of the payload. Apicurio clients and SerDe libraries can
be configured to not put the schema id in the headers in which case there
is a magic byte followed by an 8 byte schema at the start of the payload.
In the deserialization case, we would not need to look at the headers –
though the encoding is also in the headers.

That sounds like it would fit in better, I assume we cannot just take that
approach?

Thanks for the discussion. I am not 100% happy with the solution but I
cannot offer a better option. I would be interested to hear if others have
any suggestions. Playing devil's advocate against myself, we pass maps
around to configure connectors so it is not too far away from that.

Thanks,
Danny


On Fri, May 24, 2024 at 2:23 PM David Radley <da...@uk.ibm.com>
wrote:

> Hi Danny,
> No worries, thanks for replying. I have working prototype code that is
> being reviewed. It needs some cleaning up and more complete testing before
> it is ready, but will give you the general idea [1][2] to help to assess
> this approach.
>
>
> I am curious what you mean by abused. I guess the approaches are between
> generic map, mechanism vs a more particular more granular things being
> passed that might be used by another connector.
>
> Your first question:
> “how would this work if the schema ID is not in the Kafka headers, as
> hinted to in the FLIP "usually the global ID in a Kafka header"?
>
> For Apicurio it can be sent at the start of the payload like Confluent
> Avro does. Confluent Avro have a magic byte followed by 4 bytes of schema
> id, at the start of the payload. Apicurio clients and SerDe libraries can
> be configured to not put the schema id in the headers in which case there
> is a magic byte followed by an 8 byte schema at the start of the payload.
> In the deserialization case, we would not need to look at the headers –
> though the encoding is also in the headers.
>
> Your second question:
> “I am wondering if there are any other instances where the source would be
> aware of the schema ID and pass it through in this way?
> ”
> The examples I can think of are:
> - Avro can send the complete schema in a header, this is not recommended
> but in theory fits the need for a message payload to require something else
> to get the structure.
> - I see [2] that Apicurio Protobuf uses headers.
> - it might be that other message queuing projects like Rabbit MQ would
> need this to be able to support Apicurio Avro & protobuf.
>
> Kind regards, David,
>
>
>
>
> [1] https://github.com/apache/flink/pull/24715
> [2] https://github.com/apache/flink-connector-kafka/pull/99
> [3]
> https://www.apicur.io/registry/docs/apicurio-registry/2.5.x/getting-started/assembly-configuring-kafka-client-serdes.html#registry-serdes-types-json_registry
>
>
>
>
>
>
>
>
>
>
>
>
> From: Danny Cranmer <da...@apache.org>
> Date: Friday, 24 May 2024 at 12:22
> To: dev@flink.apache.org <de...@flink.apache.org>
> Subject: [EXTERNAL] Re: [DISCUSS] FLIP-XXX Apicurio-avro format
> Hello,
>
> Apologies, I am on vacation and have limited access to email.
>
> I can see the logic here and why you ended up where you did. I can also see
> there are other useful metadata fields that we might want to pass through,
> which might result in this Map being abused (Kafka Topic, Kinesis Shard,
> etc).
>
> I have a follow up question, how would this work if the schema ID is not in
> the Kafka headers, as hinted to in the FLIP "usually the global ID in a
> Kafka header"? I am wondering if there are any other instances where the
> source would be aware of the schema ID and pass it through in this way?
>
> Thanks,
> Danny
>
>
>
> On Wed, May 22, 2024 at 3:43 PM David Radley <da...@uk.ibm.com>
> wrote:
>
> > Hi Danny,
> > Did you have a chance you have a look at my responses to your feedback? I
> > am hoping to keep the momentum going on this one,   kind regards, David.
> >
> >
> > From: David Radley <da...@uk.ibm.com>
> > Date: Tuesday, 14 May 2024 at 17:21
> > To: dev@flink.apache.org <de...@flink.apache.org>
> > Subject: [EXTERNAL] [DISCUSS] FLIP-XXX Apicurio-avro format
> > Hi Danny,
> >
> > Thank you very much for the feedback and your support. I have copied your
> > feedback from the VOTE thread to this discussion thread, so we can
> continue
> > our discussions off the VOTE thread.
> >
> >
> >
> > Your feedback:
> >
> > Thanks for Driving this David. I am +1 for adding support for the new
> >
> > format, however have some questions/suggestions on the details.
> >
> >
> >
> > 1. Passing around Map<String, Object> additionalInputProperties feels a
> bit
> >
> > dirty. It looks like this is mainly for the Kafka connector. This
> connector
> >
> > already has a de/serialization schema extension to access record
> >
> > headers, KafkaRecordDeserializationSchema [1], can we use this instead?
> >
> > 2. Can you elaborate why we need to change the SchemaCoder interface?
> Again
> >
> > I am not a fan of adding these Map parameters
> >
> > 3. I assume this integration will go into the core Flink repo under
> >
> > flink-formats [2], and not be a separate repository like the connectors?
> >
> >
> >
> > My response:
> >
> > Addressing 1. and 2.
> >
> > I agree that sending maps around is a bit dirty. If we can see a better
> > way that would be great. I was looking for a way to pass this kafka
> header
> > information in a non-Kafka way - the most obvious way I could think was
> as
> > a map. Here are the main considerations I saw, if I have missed anything
> or
> > could improve something I would be grateful for any further feedback.
> >
> >
> >
> >   *   I see KafkaRecordDeserializationSchema is a Kafka interface that
> > works at the Kafka record level (so includes the headers). We need a
> > mechanism to send over the headers from the Kafka record to Flink
> >   *   Flink core is not aware of Kafka headers, and I did not want to add
> > a Kafka dependancy to core flink.
> >   *   The formats are stateless so it did not appear to be in fitting
> with
> > the Flink architecture to pass through header information to stash in
> state
> > in the format waiting for the deserialise to be subsequently called to
> pick
> > up the header information.
> >   *   We could have used Thread local storage to stash the header
> content,
> > but this would be extra state to manage; and this would seem like an
> > obtrusive change.
> >   *   The SchemaCoder deserialise is where Confluent Avro gets the schema
> > id from the payload, so it can lookup the schema. In line with this
> > approach it made sense to extend the deserialise so it had the header
> > contents so the Apicurio Avro format could lookup the schema.
> >   *   I did not want to have Apicurio specific logic in the Kafka
> > connector, if we did we could pull out the appropriate headers and only
> > send over the schema ids.
> >   *   For deserialise, the schema id we are interested in is the one in
> > the Kafka headers on the message and is for the writer schema (an Avro
> > format concept) currently used by the confluent-avro format in
> deserialize.
> >   *   For serialise the schema ids need to be obtained from apicurio then
> > passed through to Kafka.
> >   *   For serialise there is existing logic around handling the metadata
> > which includes passing the headers. But the presence of the metadata
> would
> > imply we have a metadata column. Maybe a change to the metadata mechanism
> > may have allowed to use to pass the headers, but not create a metadata
> > column; instead I pass through the additional headers in a map to be
> > appended.
> >
> >
> >
> > 3.
> >
> > Yes this integration will go into the core Flink repo under
> >
> > flink-formats and sit next to the confluent-avro format. The Avro format
> > has the concept of a Registry and drives the confluent-avro format. The
> > Apicurio Avro format will use the same approach.
> >
> > Unless otherwise stated above:
> >
> > IBM United Kingdom Limited
> > Registered in England and Wales with number 741598
> > Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU
> >
> > Unless otherwise stated above:
> >
> > IBM United Kingdom Limited
> > Registered in England and Wales with number 741598
> > Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU
> >
>
> Unless otherwise stated above:
>
> IBM United Kingdom Limited
> Registered in England and Wales with number 741598
> Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU
>

Unless otherwise stated above:

IBM United Kingdom Limited
Registered in England and Wales with number 741598
Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU

Unless otherwise stated above:

IBM United Kingdom Limited
Registered in England and Wales with number 741598
Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU

RE: [DISCUSS] FLIP-XXX Apicurio-avro format

Posted by David Radley <da...@uk.ibm.com>.
Hi Danny,
Thank you for your feedback on this.

I agree that using maps has pros and cons. The maps are flexible, but do require the sender and receiver to know what is in the map.

When you say “That sounds like it would fit in better, I assume we cannot just take that approach?” The motivation behind this Flip is to support the headers which is the usual way that Apicurio runs. We will support the “schema id in the payload” as well.

I agree with you when you say “ I am not 100% happy with the solution but I
cannot offer a better option.” – this is a pragmatic way we have found to solve this issue. I am open to any suggestions to improve this as well.

If we are going with the maps design (which is the best we have at the moment) ; it would be good to have the Flink core changes in base Flink version 2.0 as this would mean we do not need to use reflection in a Flink Kafka version 2 connector to work out if the runtime Flink has the new methods.

At this stage we only have one committer (yourself) backing this. Do you know of other 2 committers who would support this Flip?

     Kind regards, David.



From: Danny Cranmer <da...@apache.org>
Date: Friday, 24 May 2024 at 19:32
To: dev@flink.apache.org <de...@flink.apache.org>
Subject: [EXTERNAL] Re: [DISCUSS] FLIP-XXX Apicurio-avro format
Hello,

> I am curious what you mean by abused.

I just meant we will end up adding more and more fields to this map over
time, and it may be hard to undo.

> For Apicurio it can be sent at the start of the payload like Confluent
Avro does. Confluent Avro have a magic byte followed by 4 bytes of schema
id, at the start of the payload. Apicurio clients and SerDe libraries can
be configured to not put the schema id in the headers in which case there
is a magic byte followed by an 8 byte schema at the start of the payload.
In the deserialization case, we would not need to look at the headers –
though the encoding is also in the headers.

That sounds like it would fit in better, I assume we cannot just take that
approach?

Thanks for the discussion. I am not 100% happy with the solution but I
cannot offer a better option. I would be interested to hear if others have
any suggestions. Playing devil's advocate against myself, we pass maps
around to configure connectors so it is not too far away from that.

Thanks,
Danny


On Fri, May 24, 2024 at 2:23 PM David Radley <da...@uk.ibm.com>
wrote:

> Hi Danny,
> No worries, thanks for replying. I have working prototype code that is
> being reviewed. It needs some cleaning up and more complete testing before
> it is ready, but will give you the general idea [1][2] to help to assess
> this approach.
>
>
> I am curious what you mean by abused. I guess the approaches are between
> generic map, mechanism vs a more particular more granular things being
> passed that might be used by another connector.
>
> Your first question:
> “how would this work if the schema ID is not in the Kafka headers, as
> hinted to in the FLIP "usually the global ID in a Kafka header"?
>
> For Apicurio it can be sent at the start of the payload like Confluent
> Avro does. Confluent Avro have a magic byte followed by 4 bytes of schema
> id, at the start of the payload. Apicurio clients and SerDe libraries can
> be configured to not put the schema id in the headers in which case there
> is a magic byte followed by an 8 byte schema at the start of the payload.
> In the deserialization case, we would not need to look at the headers –
> though the encoding is also in the headers.
>
> Your second question:
> “I am wondering if there are any other instances where the source would be
> aware of the schema ID and pass it through in this way?
> ”
> The examples I can think of are:
> - Avro can send the complete schema in a header, this is not recommended
> but in theory fits the need for a message payload to require something else
> to get the structure.
> - I see [2] that Apicurio Protobuf uses headers.
> - it might be that other message queuing projects like Rabbit MQ would
> need this to be able to support Apicurio Avro & protobuf.
>
> Kind regards, David,
>
>
>
>
> [1] https://github.com/apache/flink/pull/24715
> [2] https://github.com/apache/flink-connector-kafka/pull/99
> [3]
> https://www.apicur.io/registry/docs/apicurio-registry/2.5.x/getting-started/assembly-configuring-kafka-client-serdes.html#registry-serdes-types-json_registry
>
>
>
>
>
>
>
>
>
>
>
>
> From: Danny Cranmer <da...@apache.org>
> Date: Friday, 24 May 2024 at 12:22
> To: dev@flink.apache.org <de...@flink.apache.org>
> Subject: [EXTERNAL] Re: [DISCUSS] FLIP-XXX Apicurio-avro format
> Hello,
>
> Apologies, I am on vacation and have limited access to email.
>
> I can see the logic here and why you ended up where you did. I can also see
> there are other useful metadata fields that we might want to pass through,
> which might result in this Map being abused (Kafka Topic, Kinesis Shard,
> etc).
>
> I have a follow up question, how would this work if the schema ID is not in
> the Kafka headers, as hinted to in the FLIP "usually the global ID in a
> Kafka header"? I am wondering if there are any other instances where the
> source would be aware of the schema ID and pass it through in this way?
>
> Thanks,
> Danny
>
>
>
> On Wed, May 22, 2024 at 3:43 PM David Radley <da...@uk.ibm.com>
> wrote:
>
> > Hi Danny,
> > Did you have a chance you have a look at my responses to your feedback? I
> > am hoping to keep the momentum going on this one,   kind regards, David.
> >
> >
> > From: David Radley <da...@uk.ibm.com>
> > Date: Tuesday, 14 May 2024 at 17:21
> > To: dev@flink.apache.org <de...@flink.apache.org>
> > Subject: [EXTERNAL] [DISCUSS] FLIP-XXX Apicurio-avro format
> > Hi Danny,
> >
> > Thank you very much for the feedback and your support. I have copied your
> > feedback from the VOTE thread to this discussion thread, so we can
> continue
> > our discussions off the VOTE thread.
> >
> >
> >
> > Your feedback:
> >
> > Thanks for Driving this David. I am +1 for adding support for the new
> >
> > format, however have some questions/suggestions on the details.
> >
> >
> >
> > 1. Passing around Map<String, Object> additionalInputProperties feels a
> bit
> >
> > dirty. It looks like this is mainly for the Kafka connector. This
> connector
> >
> > already has a de/serialization schema extension to access record
> >
> > headers, KafkaRecordDeserializationSchema [1], can we use this instead?
> >
> > 2. Can you elaborate why we need to change the SchemaCoder interface?
> Again
> >
> > I am not a fan of adding these Map parameters
> >
> > 3. I assume this integration will go into the core Flink repo under
> >
> > flink-formats [2], and not be a separate repository like the connectors?
> >
> >
> >
> > My response:
> >
> > Addressing 1. and 2.
> >
> > I agree that sending maps around is a bit dirty. If we can see a better
> > way that would be great. I was looking for a way to pass this kafka
> header
> > information in a non-Kafka way - the most obvious way I could think was
> as
> > a map. Here are the main considerations I saw, if I have missed anything
> or
> > could improve something I would be grateful for any further feedback.
> >
> >
> >
> >   *   I see KafkaRecordDeserializationSchema is a Kafka interface that
> > works at the Kafka record level (so includes the headers). We need a
> > mechanism to send over the headers from the Kafka record to Flink
> >   *   Flink core is not aware of Kafka headers, and I did not want to add
> > a Kafka dependancy to core flink.
> >   *   The formats are stateless so it did not appear to be in fitting
> with
> > the Flink architecture to pass through header information to stash in
> state
> > in the format waiting for the deserialise to be subsequently called to
> pick
> > up the header information.
> >   *   We could have used Thread local storage to stash the header
> content,
> > but this would be extra state to manage; and this would seem like an
> > obtrusive change.
> >   *   The SchemaCoder deserialise is where Confluent Avro gets the schema
> > id from the payload, so it can lookup the schema. In line with this
> > approach it made sense to extend the deserialise so it had the header
> > contents so the Apicurio Avro format could lookup the schema.
> >   *   I did not want to have Apicurio specific logic in the Kafka
> > connector, if we did we could pull out the appropriate headers and only
> > send over the schema ids.
> >   *   For deserialise, the schema id we are interested in is the one in
> > the Kafka headers on the message and is for the writer schema (an Avro
> > format concept) currently used by the confluent-avro format in
> deserialize.
> >   *   For serialise the schema ids need to be obtained from apicurio then
> > passed through to Kafka.
> >   *   For serialise there is existing logic around handling the metadata
> > which includes passing the headers. But the presence of the metadata
> would
> > imply we have a metadata column. Maybe a change to the metadata mechanism
> > may have allowed to use to pass the headers, but not create a metadata
> > column; instead I pass through the additional headers in a map to be
> > appended.
> >
> >
> >
> > 3.
> >
> > Yes this integration will go into the core Flink repo under
> >
> > flink-formats and sit next to the confluent-avro format. The Avro format
> > has the concept of a Registry and drives the confluent-avro format. The
> > Apicurio Avro format will use the same approach.
> >
> > Unless otherwise stated above:
> >
> > IBM United Kingdom Limited
> > Registered in England and Wales with number 741598
> > Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU
> >
> > Unless otherwise stated above:
> >
> > IBM United Kingdom Limited
> > Registered in England and Wales with number 741598
> > Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU
> >
>
> Unless otherwise stated above:
>
> IBM United Kingdom Limited
> Registered in England and Wales with number 741598
> Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU
>

Unless otherwise stated above:

IBM United Kingdom Limited
Registered in England and Wales with number 741598
Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU

Re: [DISCUSS] FLIP-XXX Apicurio-avro format

Posted by Danny Cranmer <da...@apache.org>.
Hello,

> I am curious what you mean by abused.

I just meant we will end up adding more and more fields to this map over
time, and it may be hard to undo.

> For Apicurio it can be sent at the start of the payload like Confluent
Avro does. Confluent Avro have a magic byte followed by 4 bytes of schema
id, at the start of the payload. Apicurio clients and SerDe libraries can
be configured to not put the schema id in the headers in which case there
is a magic byte followed by an 8 byte schema at the start of the payload.
In the deserialization case, we would not need to look at the headers –
though the encoding is also in the headers.

That sounds like it would fit in better, I assume we cannot just take that
approach?

Thanks for the discussion. I am not 100% happy with the solution but I
cannot offer a better option. I would be interested to hear if others have
any suggestions. Playing devil's advocate against myself, we pass maps
around to configure connectors so it is not too far away from that.

Thanks,
Danny


On Fri, May 24, 2024 at 2:23 PM David Radley <da...@uk.ibm.com>
wrote:

> Hi Danny,
> No worries, thanks for replying. I have working prototype code that is
> being reviewed. It needs some cleaning up and more complete testing before
> it is ready, but will give you the general idea [1][2] to help to assess
> this approach.
>
>
> I am curious what you mean by abused. I guess the approaches are between
> generic map, mechanism vs a more particular more granular things being
> passed that might be used by another connector.
>
> Your first question:
> “how would this work if the schema ID is not in the Kafka headers, as
> hinted to in the FLIP "usually the global ID in a Kafka header"?
>
> For Apicurio it can be sent at the start of the payload like Confluent
> Avro does. Confluent Avro have a magic byte followed by 4 bytes of schema
> id, at the start of the payload. Apicurio clients and SerDe libraries can
> be configured to not put the schema id in the headers in which case there
> is a magic byte followed by an 8 byte schema at the start of the payload.
> In the deserialization case, we would not need to look at the headers –
> though the encoding is also in the headers.
>
> Your second question:
> “I am wondering if there are any other instances where the source would be
> aware of the schema ID and pass it through in this way?
> ”
> The examples I can think of are:
> - Avro can send the complete schema in a header, this is not recommended
> but in theory fits the need for a message payload to require something else
> to get the structure.
> - I see [2] that Apicurio Protobuf uses headers.
> - it might be that other message queuing projects like Rabbit MQ would
> need this to be able to support Apicurio Avro & protobuf.
>
> Kind regards, David,
>
>
>
>
> [1] https://github.com/apache/flink/pull/24715
> [2] https://github.com/apache/flink-connector-kafka/pull/99
> [3]
> https://www.apicur.io/registry/docs/apicurio-registry/2.5.x/getting-started/assembly-configuring-kafka-client-serdes.html#registry-serdes-types-json_registry
>
>
>
>
>
>
>
>
>
>
>
>
> From: Danny Cranmer <da...@apache.org>
> Date: Friday, 24 May 2024 at 12:22
> To: dev@flink.apache.org <de...@flink.apache.org>
> Subject: [EXTERNAL] Re: [DISCUSS] FLIP-XXX Apicurio-avro format
> Hello,
>
> Apologies, I am on vacation and have limited access to email.
>
> I can see the logic here and why you ended up where you did. I can also see
> there are other useful metadata fields that we might want to pass through,
> which might result in this Map being abused (Kafka Topic, Kinesis Shard,
> etc).
>
> I have a follow up question, how would this work if the schema ID is not in
> the Kafka headers, as hinted to in the FLIP "usually the global ID in a
> Kafka header"? I am wondering if there are any other instances where the
> source would be aware of the schema ID and pass it through in this way?
>
> Thanks,
> Danny
>
>
>
> On Wed, May 22, 2024 at 3:43 PM David Radley <da...@uk.ibm.com>
> wrote:
>
> > Hi Danny,
> > Did you have a chance you have a look at my responses to your feedback? I
> > am hoping to keep the momentum going on this one,   kind regards, David.
> >
> >
> > From: David Radley <da...@uk.ibm.com>
> > Date: Tuesday, 14 May 2024 at 17:21
> > To: dev@flink.apache.org <de...@flink.apache.org>
> > Subject: [EXTERNAL] [DISCUSS] FLIP-XXX Apicurio-avro format
> > Hi Danny,
> >
> > Thank you very much for the feedback and your support. I have copied your
> > feedback from the VOTE thread to this discussion thread, so we can
> continue
> > our discussions off the VOTE thread.
> >
> >
> >
> > Your feedback:
> >
> > Thanks for Driving this David. I am +1 for adding support for the new
> >
> > format, however have some questions/suggestions on the details.
> >
> >
> >
> > 1. Passing around Map<String, Object> additionalInputProperties feels a
> bit
> >
> > dirty. It looks like this is mainly for the Kafka connector. This
> connector
> >
> > already has a de/serialization schema extension to access record
> >
> > headers, KafkaRecordDeserializationSchema [1], can we use this instead?
> >
> > 2. Can you elaborate why we need to change the SchemaCoder interface?
> Again
> >
> > I am not a fan of adding these Map parameters
> >
> > 3. I assume this integration will go into the core Flink repo under
> >
> > flink-formats [2], and not be a separate repository like the connectors?
> >
> >
> >
> > My response:
> >
> > Addressing 1. and 2.
> >
> > I agree that sending maps around is a bit dirty. If we can see a better
> > way that would be great. I was looking for a way to pass this kafka
> header
> > information in a non-Kafka way - the most obvious way I could think was
> as
> > a map. Here are the main considerations I saw, if I have missed anything
> or
> > could improve something I would be grateful for any further feedback.
> >
> >
> >
> >   *   I see KafkaRecordDeserializationSchema is a Kafka interface that
> > works at the Kafka record level (so includes the headers). We need a
> > mechanism to send over the headers from the Kafka record to Flink
> >   *   Flink core is not aware of Kafka headers, and I did not want to add
> > a Kafka dependancy to core flink.
> >   *   The formats are stateless so it did not appear to be in fitting
> with
> > the Flink architecture to pass through header information to stash in
> state
> > in the format waiting for the deserialise to be subsequently called to
> pick
> > up the header information.
> >   *   We could have used Thread local storage to stash the header
> content,
> > but this would be extra state to manage; and this would seem like an
> > obtrusive change.
> >   *   The SchemaCoder deserialise is where Confluent Avro gets the schema
> > id from the payload, so it can lookup the schema. In line with this
> > approach it made sense to extend the deserialise so it had the header
> > contents so the Apicurio Avro format could lookup the schema.
> >   *   I did not want to have Apicurio specific logic in the Kafka
> > connector, if we did we could pull out the appropriate headers and only
> > send over the schema ids.
> >   *   For deserialise, the schema id we are interested in is the one in
> > the Kafka headers on the message and is for the writer schema (an Avro
> > format concept) currently used by the confluent-avro format in
> deserialize.
> >   *   For serialise the schema ids need to be obtained from apicurio then
> > passed through to Kafka.
> >   *   For serialise there is existing logic around handling the metadata
> > which includes passing the headers. But the presence of the metadata
> would
> > imply we have a metadata column. Maybe a change to the metadata mechanism
> > may have allowed to use to pass the headers, but not create a metadata
> > column; instead I pass through the additional headers in a map to be
> > appended.
> >
> >
> >
> > 3.
> >
> > Yes this integration will go into the core Flink repo under
> >
> > flink-formats and sit next to the confluent-avro format. The Avro format
> > has the concept of a Registry and drives the confluent-avro format. The
> > Apicurio Avro format will use the same approach.
> >
> > Unless otherwise stated above:
> >
> > IBM United Kingdom Limited
> > Registered in England and Wales with number 741598
> > Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU
> >
> > Unless otherwise stated above:
> >
> > IBM United Kingdom Limited
> > Registered in England and Wales with number 741598
> > Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU
> >
>
> Unless otherwise stated above:
>
> IBM United Kingdom Limited
> Registered in England and Wales with number 741598
> Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU
>

RE: [DISCUSS] FLIP-XXX Apicurio-avro format

Posted by David Radley <da...@uk.ibm.com>.
Hi Danny,
No worries, thanks for replying. I have working prototype code that is being reviewed. It needs some cleaning up and more complete testing before it is ready, but will give you the general idea [1][2] to help to assess this approach.


I am curious what you mean by abused. I guess the approaches are between generic map, mechanism vs a more particular more granular things being passed that might be used by another connector.

Your first question:
“how would this work if the schema ID is not in the Kafka headers, as hinted to in the FLIP "usually the global ID in a Kafka header"?

For Apicurio it can be sent at the start of the payload like Confluent Avro does. Confluent Avro have a magic byte followed by 4 bytes of schema id, at the start of the payload. Apicurio clients and SerDe libraries can be configured to not put the schema id in the headers in which case there is a magic byte followed by an 8 byte schema at the start of the payload. In the deserialization case, we would not need to look at the headers – though the encoding is also in the headers.

Your second question:
“I am wondering if there are any other instances where the source would be aware of the schema ID and pass it through in this way?
”
The examples I can think of are:
- Avro can send the complete schema in a header, this is not recommended but in theory fits the need for a message payload to require something else to get the structure.
- I see [2] that Apicurio Protobuf uses headers.
- it might be that other message queuing projects like Rabbit MQ would need this to be able to support Apicurio Avro & protobuf.

Kind regards, David,




[1] https://github.com/apache/flink/pull/24715
[2] https://github.com/apache/flink-connector-kafka/pull/99
[3] https://www.apicur.io/registry/docs/apicurio-registry/2.5.x/getting-started/assembly-configuring-kafka-client-serdes.html#registry-serdes-types-json_registry












From: Danny Cranmer <da...@apache.org>
Date: Friday, 24 May 2024 at 12:22
To: dev@flink.apache.org <de...@flink.apache.org>
Subject: [EXTERNAL] Re: [DISCUSS] FLIP-XXX Apicurio-avro format
Hello,

Apologies, I am on vacation and have limited access to email.

I can see the logic here and why you ended up where you did. I can also see
there are other useful metadata fields that we might want to pass through,
which might result in this Map being abused (Kafka Topic, Kinesis Shard,
etc).

I have a follow up question, how would this work if the schema ID is not in
the Kafka headers, as hinted to in the FLIP "usually the global ID in a
Kafka header"? I am wondering if there are any other instances where the
source would be aware of the schema ID and pass it through in this way?

Thanks,
Danny



On Wed, May 22, 2024 at 3:43 PM David Radley <da...@uk.ibm.com>
wrote:

> Hi Danny,
> Did you have a chance you have a look at my responses to your feedback? I
> am hoping to keep the momentum going on this one,   kind regards, David.
>
>
> From: David Radley <da...@uk.ibm.com>
> Date: Tuesday, 14 May 2024 at 17:21
> To: dev@flink.apache.org <de...@flink.apache.org>
> Subject: [EXTERNAL] [DISCUSS] FLIP-XXX Apicurio-avro format
> Hi Danny,
>
> Thank you very much for the feedback and your support. I have copied your
> feedback from the VOTE thread to this discussion thread, so we can continue
> our discussions off the VOTE thread.
>
>
>
> Your feedback:
>
> Thanks for Driving this David. I am +1 for adding support for the new
>
> format, however have some questions/suggestions on the details.
>
>
>
> 1. Passing around Map<String, Object> additionalInputProperties feels a bit
>
> dirty. It looks like this is mainly for the Kafka connector. This connector
>
> already has a de/serialization schema extension to access record
>
> headers, KafkaRecordDeserializationSchema [1], can we use this instead?
>
> 2. Can you elaborate why we need to change the SchemaCoder interface? Again
>
> I am not a fan of adding these Map parameters
>
> 3. I assume this integration will go into the core Flink repo under
>
> flink-formats [2], and not be a separate repository like the connectors?
>
>
>
> My response:
>
> Addressing 1. and 2.
>
> I agree that sending maps around is a bit dirty. If we can see a better
> way that would be great. I was looking for a way to pass this kafka header
> information in a non-Kafka way - the most obvious way I could think was as
> a map. Here are the main considerations I saw, if I have missed anything or
> could improve something I would be grateful for any further feedback.
>
>
>
>   *   I see KafkaRecordDeserializationSchema is a Kafka interface that
> works at the Kafka record level (so includes the headers). We need a
> mechanism to send over the headers from the Kafka record to Flink
>   *   Flink core is not aware of Kafka headers, and I did not want to add
> a Kafka dependancy to core flink.
>   *   The formats are stateless so it did not appear to be in fitting with
> the Flink architecture to pass through header information to stash in state
> in the format waiting for the deserialise to be subsequently called to pick
> up the header information.
>   *   We could have used Thread local storage to stash the header content,
> but this would be extra state to manage; and this would seem like an
> obtrusive change.
>   *   The SchemaCoder deserialise is where Confluent Avro gets the schema
> id from the payload, so it can lookup the schema. In line with this
> approach it made sense to extend the deserialise so it had the header
> contents so the Apicurio Avro format could lookup the schema.
>   *   I did not want to have Apicurio specific logic in the Kafka
> connector, if we did we could pull out the appropriate headers and only
> send over the schema ids.
>   *   For deserialise, the schema id we are interested in is the one in
> the Kafka headers on the message and is for the writer schema (an Avro
> format concept) currently used by the confluent-avro format in deserialize.
>   *   For serialise the schema ids need to be obtained from apicurio then
> passed through to Kafka.
>   *   For serialise there is existing logic around handling the metadata
> which includes passing the headers. But the presence of the metadata would
> imply we have a metadata column. Maybe a change to the metadata mechanism
> may have allowed to use to pass the headers, but not create a metadata
> column; instead I pass through the additional headers in a map to be
> appended.
>
>
>
> 3.
>
> Yes this integration will go into the core Flink repo under
>
> flink-formats and sit next to the confluent-avro format. The Avro format
> has the concept of a Registry and drives the confluent-avro format. The
> Apicurio Avro format will use the same approach.
>
> Unless otherwise stated above:
>
> IBM United Kingdom Limited
> Registered in England and Wales with number 741598
> Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU
>
> Unless otherwise stated above:
>
> IBM United Kingdom Limited
> Registered in England and Wales with number 741598
> Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU
>

Unless otherwise stated above:

IBM United Kingdom Limited
Registered in England and Wales with number 741598
Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU

Re: [DISCUSS] FLIP-XXX Apicurio-avro format

Posted by Danny Cranmer <da...@apache.org>.
Hello,

Apologies, I am on vacation and have limited access to email.

I can see the logic here and why you ended up where you did. I can also see
there are other useful metadata fields that we might want to pass through,
which might result in this Map being abused (Kafka Topic, Kinesis Shard,
etc).

I have a follow up question, how would this work if the schema ID is not in
the Kafka headers, as hinted to in the FLIP "usually the global ID in a
Kafka header"? I am wondering if there are any other instances where the
source would be aware of the schema ID and pass it through in this way?

Thanks,
Danny



On Wed, May 22, 2024 at 3:43 PM David Radley <da...@uk.ibm.com>
wrote:

> Hi Danny,
> Did you have a chance you have a look at my responses to your feedback? I
> am hoping to keep the momentum going on this one,   kind regards, David.
>
>
> From: David Radley <da...@uk.ibm.com>
> Date: Tuesday, 14 May 2024 at 17:21
> To: dev@flink.apache.org <de...@flink.apache.org>
> Subject: [EXTERNAL] [DISCUSS] FLIP-XXX Apicurio-avro format
> Hi Danny,
>
> Thank you very much for the feedback and your support. I have copied your
> feedback from the VOTE thread to this discussion thread, so we can continue
> our discussions off the VOTE thread.
>
>
>
> Your feedback:
>
> Thanks for Driving this David. I am +1 for adding support for the new
>
> format, however have some questions/suggestions on the details.
>
>
>
> 1. Passing around Map<String, Object> additionalInputProperties feels a bit
>
> dirty. It looks like this is mainly for the Kafka connector. This connector
>
> already has a de/serialization schema extension to access record
>
> headers, KafkaRecordDeserializationSchema [1], can we use this instead?
>
> 2. Can you elaborate why we need to change the SchemaCoder interface? Again
>
> I am not a fan of adding these Map parameters
>
> 3. I assume this integration will go into the core Flink repo under
>
> flink-formats [2], and not be a separate repository like the connectors?
>
>
>
> My response:
>
> Addressing 1. and 2.
>
> I agree that sending maps around is a bit dirty. If we can see a better
> way that would be great. I was looking for a way to pass this kafka header
> information in a non-Kafka way - the most obvious way I could think was as
> a map. Here are the main considerations I saw, if I have missed anything or
> could improve something I would be grateful for any further feedback.
>
>
>
>   *   I see KafkaRecordDeserializationSchema is a Kafka interface that
> works at the Kafka record level (so includes the headers). We need a
> mechanism to send over the headers from the Kafka record to Flink
>   *   Flink core is not aware of Kafka headers, and I did not want to add
> a Kafka dependancy to core flink.
>   *   The formats are stateless so it did not appear to be in fitting with
> the Flink architecture to pass through header information to stash in state
> in the format waiting for the deserialise to be subsequently called to pick
> up the header information.
>   *   We could have used Thread local storage to stash the header content,
> but this would be extra state to manage; and this would seem like an
> obtrusive change.
>   *   The SchemaCoder deserialise is where Confluent Avro gets the schema
> id from the payload, so it can lookup the schema. In line with this
> approach it made sense to extend the deserialise so it had the header
> contents so the Apicurio Avro format could lookup the schema.
>   *   I did not want to have Apicurio specific logic in the Kafka
> connector, if we did we could pull out the appropriate headers and only
> send over the schema ids.
>   *   For deserialise, the schema id we are interested in is the one in
> the Kafka headers on the message and is for the writer schema (an Avro
> format concept) currently used by the confluent-avro format in deserialize.
>   *   For serialise the schema ids need to be obtained from apicurio then
> passed through to Kafka.
>   *   For serialise there is existing logic around handling the metadata
> which includes passing the headers. But the presence of the metadata would
> imply we have a metadata column. Maybe a change to the metadata mechanism
> may have allowed to use to pass the headers, but not create a metadata
> column; instead I pass through the additional headers in a map to be
> appended.
>
>
>
> 3.
>
> Yes this integration will go into the core Flink repo under
>
> flink-formats and sit next to the confluent-avro format. The Avro format
> has the concept of a Registry and drives the confluent-avro format. The
> Apicurio Avro format will use the same approach.
>
> Unless otherwise stated above:
>
> IBM United Kingdom Limited
> Registered in England and Wales with number 741598
> Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU
>
> Unless otherwise stated above:
>
> IBM United Kingdom Limited
> Registered in England and Wales with number 741598
> Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU
>

Re: [DISCUSS] FLIP-XXX Apicurio-avro format

Posted by David Radley <da...@uk.ibm.com>.
Hi Danny,
Did you have a chance you have a look at my responses to your feedback? I am hoping to keep the momentum going on this one,   kind regards, David.


From: David Radley <da...@uk.ibm.com>
Date: Tuesday, 14 May 2024 at 17:21
To: dev@flink.apache.org <de...@flink.apache.org>
Subject: [EXTERNAL] [DISCUSS] FLIP-XXX Apicurio-avro format
Hi Danny,

Thank you very much for the feedback and your support. I have copied your feedback from the VOTE thread to this discussion thread, so we can continue our discussions off the VOTE thread.



Your feedback:

Thanks for Driving this David. I am +1 for adding support for the new

format, however have some questions/suggestions on the details.



1. Passing around Map<String, Object> additionalInputProperties feels a bit

dirty. It looks like this is mainly for the Kafka connector. This connector

already has a de/serialization schema extension to access record

headers, KafkaRecordDeserializationSchema [1], can we use this instead?

2. Can you elaborate why we need to change the SchemaCoder interface? Again

I am not a fan of adding these Map parameters

3. I assume this integration will go into the core Flink repo under

flink-formats [2], and not be a separate repository like the connectors?



My response:

Addressing 1. and 2.

I agree that sending maps around is a bit dirty. If we can see a better way that would be great. I was looking for a way to pass this kafka header information in a non-Kafka way - the most obvious way I could think was as a map. Here are the main considerations I saw, if I have missed anything or could improve something I would be grateful for any further feedback.



  *   I see KafkaRecordDeserializationSchema is a Kafka interface that works at the Kafka record level (so includes the headers). We need a mechanism to send over the headers from the Kafka record to Flink
  *   Flink core is not aware of Kafka headers, and I did not want to add a Kafka dependancy to core flink.
  *   The formats are stateless so it did not appear to be in fitting with the Flink architecture to pass through header information to stash in state in the format waiting for the deserialise to be subsequently called to pick up the header information.
  *   We could have used Thread local storage to stash the header content, but this would be extra state to manage; and this would seem like an obtrusive change.
  *   The SchemaCoder deserialise is where Confluent Avro gets the schema id from the payload, so it can lookup the schema. In line with this approach it made sense to extend the deserialise so it had the header contents so the Apicurio Avro format could lookup the schema.
  *   I did not want to have Apicurio specific logic in the Kafka connector, if we did we could pull out the appropriate headers and only send over the schema ids.
  *   For deserialise, the schema id we are interested in is the one in the Kafka headers on the message and is for the writer schema (an Avro format concept) currently used by the confluent-avro format in deserialize.
  *   For serialise the schema ids need to be obtained from apicurio then passed through to Kafka.
  *   For serialise there is existing logic around handling the metadata which includes passing the headers. But the presence of the metadata would imply we have a metadata column. Maybe a change to the metadata mechanism may have allowed to use to pass the headers, but not create a metadata column; instead I pass through the additional headers in a map to be appended.



3.

Yes this integration will go into the core Flink repo under

flink-formats and sit next to the confluent-avro format. The Avro format has the concept of a Registry and drives the confluent-avro format. The Apicurio Avro format will use the same approach.

Unless otherwise stated above:

IBM United Kingdom Limited
Registered in England and Wales with number 741598
Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU

Unless otherwise stated above:

IBM United Kingdom Limited
Registered in England and Wales with number 741598
Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU

[DISCUSS] FLIP-XXX Apicurio-avro format

Posted by David Radley <da...@uk.ibm.com>.
Hi Danny,

Thank you very much for the feedback and your support. I have copied your feedback from the VOTE thread to this discussion thread, so we can continue our discussions off the VOTE thread.



Your feedback:

Thanks for Driving this David. I am +1 for adding support for the new

format, however have some questions/suggestions on the details.



1. Passing around Map<String, Object> additionalInputProperties feels a bit

dirty. It looks like this is mainly for the Kafka connector. This connector

already has a de/serialization schema extension to access record

headers, KafkaRecordDeserializationSchema [1], can we use this instead?

2. Can you elaborate why we need to change the SchemaCoder interface? Again

I am not a fan of adding these Map parameters

3. I assume this integration will go into the core Flink repo under

flink-formats [2], and not be a separate repository like the connectors?



My response:

Addressing 1. and 2.

I agree that sending maps around is a bit dirty. If we can see a better way that would be great. I was looking for a way to pass this kafka header information in a non-Kafka way - the most obvious way I could think was as a map. Here are the main considerations I saw, if I have missed anything or could improve something I would be grateful for any further feedback.



  *   I see KafkaRecordDeserializationSchema is a Kafka interface that works at the Kafka record level (so includes the headers). We need a mechanism to send over the headers from the Kafka record to Flink
  *   Flink core is not aware of Kafka headers, and I did not want to add a Kafka dependancy to core flink.
  *   The formats are stateless so it did not appear to be in fitting with the Flink architecture to pass through header information to stash in state in the format waiting for the deserialise to be subsequently called to pick up the header information.
  *   We could have used Thread local storage to stash the header content, but this would be extra state to manage; and this would seem like an obtrusive change.
  *   The SchemaCoder deserialise is where Confluent Avro gets the schema id from the payload, so it can lookup the schema. In line with this approach it made sense to extend the deserialise so it had the header contents so the Apicurio Avro format could lookup the schema.
  *   I did not want to have Apicurio specific logic in the Kafka connector, if we did we could pull out the appropriate headers and only send over the schema ids.
  *   For deserialise, the schema id we are interested in is the one in the Kafka headers on the message and is for the writer schema (an Avro format concept) currently used by the confluent-avro format in deserialize.
  *   For serialise the schema ids need to be obtained from apicurio then passed through to Kafka.
  *   For serialise there is existing logic around handling the metadata which includes passing the headers. But the presence of the metadata would imply we have a metadata column. Maybe a change to the metadata mechanism may have allowed to use to pass the headers, but not create a metadata column; instead I pass through the additional headers in a map to be appended.



3.

Yes this integration will go into the core Flink repo under

flink-formats and sit next to the confluent-avro format. The Avro format has the concept of a Registry and drives the confluent-avro format. The Apicurio Avro format will use the same approach.

Unless otherwise stated above:

IBM United Kingdom Limited
Registered in England and Wales with number 741598
Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU

RE: FW: RE: [DISCUSS] FLIP-XXX Apicurio-avro format

Posted by David Radley <da...@uk.ibm.com>.
Hi Martijn,
I have started a vote thread – please could you update the Flip with the link to the vote thread,
      Kind regards, David.


From: David Radley <da...@uk.ibm.com>
Date: Thursday, 2 May 2024 at 10:39
To: dev@flink.apache.org <de...@flink.apache.org>
Subject: [EXTERNAL] RE: FW: RE: [DISCUSS] FLIP-XXX Apicurio-avro format
Fabulous, thanks Martijn 😊

From: Martijn Visser <ma...@apache.org>
Date: Thursday, 2 May 2024 at 10:08
To: dev@flink.apache.org <de...@flink.apache.org>
Subject: [EXTERNAL] Re: FW: RE: [DISCUSS] FLIP-XXX Apicurio-avro format
Done :)

On Thu, May 2, 2024 at 11:01 AM David Radley <da...@uk.ibm.com>
wrote:

> Hi Martijn,
> Thank you very much for looking at this. In response to your feedback; I
> produced a reduced version which is on this link.
>
>
> https://docs.google.com/document/d/1J1E-cE-X2H3-kw4rNjLn71OGPQk_Yl1iGX4-eCHWLgE/edit?usp=sharing
>
> The original version you have copied is a bit out-dated and verbose.
> Please could you replace the Flip with content from the above link,
>                                 Kind regards, David,
>
> From: Martijn Visser <ma...@apache.org>
> Date: Wednesday, 1 May 2024 at 16:31
> To: dev@flink.apache.org <de...@flink.apache.org>
> Subject: [EXTERNAL] Re: FW: RE: [DISCUSS] FLIP-XXX Apicurio-avro format
> Hi David,
>
> I've copied and pasted it into
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-454%3A+New+Apicurio+Avro+format
> ;
> please take a look if it's as expected.
>
> Best regards,
>
> Martijn
>
> On Wed, May 1, 2024 at 3:47 PM David Radley <da...@uk.ibm.com>
> wrote:
>
> > Hi Martijn,
> > Any news?
> >            Kind regards, David.
> >
> >
> > From: David Radley <da...@uk.ibm.com>
> > Date: Monday, 22 April 2024 at 09:48
> > To: dev@flink.apache.org <de...@flink.apache.org>
> > Subject: FW: [EXTERNAL] RE: [DISCUSS] FLIP-XXX Apicurio-avro format
> > Hi Martijn,
> > A gentle nudge, is this ok for you or one of the PMC or committers to
> > create a Flip now?
> >            Kind regards, David.
> >
> > From: David Radley <da...@uk.ibm.com>
> > Date: Monday, 15 April 2024 at 12:29
> > To: dev@flink.apache.org <de...@flink.apache.org>
> > Subject: Re: [EXTERNAL] RE: [DISCUSS] FLIP-XXX Apicurio-avro format
> > Hi Martijn,
> > Thanks for looking at this. I have used the template in a new  Google Doc
> >
> https://docs.google.com/document/d/1J1E-cE-X2H3-kw4rNjLn71OGPQk_Yl1iGX4-eCHWLgE/edit?usp=sharing
> .
> > I have significantly reduced the content in the Flip, in line with what I
> > see as the template and its usage. If this it too much or too little, I
> can
> > amend,
> >
> >     Kind regards, David.
> >
> > From: Martijn Visser <ma...@apache.org>
> > Date: Friday, 12 April 2024 at 18:11
> > To: dev@flink.apache.org <de...@flink.apache.org>
> > Subject: Re: [EXTERNAL] RE: [DISCUSS] FLIP-XXX Apicurio-avro format
> > Hi David,
> >
> > I tried, but the format wasn't as the FLIP template expects, so I ended
> up
> > needing to change the entire formatting and that was just too much work
> to
> > be honest. If you could make sure that especially the headers match with
> > the FLIP template, and that all of the contents from the FLIP template is
> > there, that would make things much easier.
> >
> > Thanks,
> >
> > Martijn
> >
> > On Fri, Apr 12, 2024 at 6:08 PM David Radley <da...@uk.ibm.com>
> > wrote:
> >
> > > Hi,
> > > A gentle nudge. Please could a committer/PMC member raise the Flip for
> > > this,
> > >       Kind regards, David.
> > >
> > >
> > > From: David Radley <da...@uk.ibm.com>
> > > Date: Monday, 8 April 2024 at 09:40
> > > To: dev@flink.apache.org <de...@flink.apache.org>
> > > Subject: [EXTERNAL] RE: [DISCUSS] FLIP-XXX Apicurio-avro format
> > > Hi,
> > > I have posted a Google Doc [0] to the mailing list for a discussion
> > thread
> > > for a Flip proposal to introduce a Apicurio-avro format. The
> discussions
> > > have been resolved, please could a committer/PMC member copy the
> contents
> > > from the Google Doc, and create a FLIP number for this,. as per the
> > process
> > > [1],
> > >       Kind regards, David.
> > > [0]
> > >
> > >
> >
> https://docs.google.com/document/d/14LWZPVFQ7F9mryJPdKXb4l32n7B0iWYkcOdEd1xTC7w/edit?usp=sharing
> > >
> > > [1]
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals#FlinkImprovementProposals-CreateyourOwnFLIP
> > >
> > > From: Jeyhun Karimov <je...@gmail.com>
> > > Date: Friday, 22 March 2024 at 13:05
> > > To: dev@flink.apache.org <de...@flink.apache.org>
> > > Subject: [EXTERNAL] Re: [DISCUSS] FLIP-XXX Apicurio-avro format
> > > Hi David,
> > >
> > > Thanks a lot for clarification.
> > > Sounds good to me.
> > >
> > > Regards,
> > > Jeyhun
> > >
> > > On Fri, Mar 22, 2024 at 10:54 AM David Radley <david_radley@uk.ibm.com
> >
> > > wrote:
> > >
> > > > Hi Jeyhun,
> > > > Thanks for your feedback.
> > > >
> > > > So for outbound messages, the message includes the global ID. We
> > register
> > > > the schema and match on the artifact id. So if the schema then
> evolved,
> > > > adding a new  version, the global ID would still be unique and the
> same
> > > > version would be targeted. If you wanted to change the Flink table
> > > > definition in line with a higher version, then you could do this –
> the
> > > > artifact id would need to match for it to use the same schema and a
> > > higher
> > > > artifact version would need to be provided. I notice that Apicurio
> has
> > > > rules around compatibility that you can configure, I suppose if we
> > > attempt
> > > > to create an artifact that breaks these rules , then the register
> > schema
> > > > will fail and the associated operation should fail (e.g. an insert).
> I
> > > have
> > > > not tried this.
> > > >
> > > >
> > > > For inbound messages, using the global id in the header – this
> targets
> > > one
> > > > version of the schema. I can create different messages on the topic
> > built
> > > > with different schema versions, and I can create different tables in
> > > Flink,
> > > > as long as the reader and writer schemas are compatible as per the
> > > >
> > >
> >
> https://github.com/apache/flink/blob/779459168c46b7b4c600ef52f99a5435f81b9048/flink-formats/flink-avro/src/main/java/org/apache/flink/formats/avro/RegistryAvroDeserializationSchema.java#L109
> > > > Then this should work.
> > > >
> > > > Does this address your question?
> > > >     Kind regards, David.
> > > >
> > > >
> > > > From: Jeyhun Karimov <je...@gmail.com>
> > > > Date: Thursday, 21 March 2024 at 21:06
> > > > To: dev@flink.apache.org <de...@flink.apache.org>
> > > > Subject: [EXTERNAL] Re: [DISCUSS] FLIP-XXX Apicurio-avro format
> > > > Hi David,
> > > >
> > > > Thanks for the FLIP. +1 for it.
> > > > I have a minor comment.
> > > >
> > > > Can you please elaborate more on mechanisms in place to ensure data
> > > > consistency and integrity, particularly in the event of schema
> > conflicts?
> > > > Since each message includes a schema ID for inbound and outbound
> > > messages,
> > > > can you elaborate more on message consistency in the context of
> schema
> > > > evolution?
> > > >
> > > > Regards,
> > > > Jeyhun
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On Wed, Mar 20, 2024 at 4:34 PM David Radley <da...@apache.org>
> > > wrote:
> > > >
> > > > > Thank you very much for your feedback Mark. I have made the changes
> > in
> > > > the
> > > > > latest google document. On reflection I agree with you that the
> > > > > globalIdPlacement format configuration should apply to the
> > > > deserialization
> > > > > as well, so it is declarative. I am also going to have a new
> > > > configuration
> > > > > option to work with content IDs as well as global IDs. In line with
> > the
> > > > > deser Apicurio IdHandler and headerHandlers.
> > > > >
> > > > >  kind regards, David.
> > > > >
> > > > >
> > > > > On 2024/03/20 15:18:37 Mark Nuttall wrote:
> > > > > > +1 to this
> > > > > >
> > > > > > A few small comments:
> > > > > >
> > > > > > Currently, if users have Avro schemas in an Apicurio Registry (an
> > > open
> > > > > source Apache 2 licensed schema registry), then the natural way to
> > work
> > > > > with those Avro flows is to use the schemas in the Apicurio
> > Repository.
> > > > > > 'those Avro flows' ... this is the first reference to flows.
> > > > > >
> > > > > > The new format will use the global Id to look up the Avro schema
> > that
> > > > > the message was written during deserialization.
> > > > > > I get the point, phrasing is awkward. Probably you're more
> > interested
> > > > in
> > > > > content than word polish at this point though.
> > > > > >
> > > > > > The Avro Schema Registry (apicurio-avro) format
> > > > > > The Confluent format is called avro-confluent; this should be
> > > > > avro-apicurio
> > > > > >
> > > > > > How to create tables with Apicurio-avro format
> > > > > > s/Apicurio-avro/avro-apicurio/g
> > > > > >
> > > > > > HEADER – globalId is put in the header
> > > > > > LEGACY– global Id is put in the message as a long
> > > > > > CONFLUENT - globalId is put in the message as an int.
> > > > > > Please could we specify 'four-byte int' and 'eight-byte long' ?
> > > > > >
> > > > > > For a Kafka source the globalId will be looked for in this order:
> > > > > > -     In the header
> > > > > > -     After a magic byte as an int
> > > > > > -     After a magic byte as a long.
> > > > > > but apicurio-avro.globalid-placement has a default value of
> HEADER
> > :
> > > > why
> > > > > do we have a search order as well? Isn't
> > > apicurio-avro.globalid-placement
> > > > > enough? Don't the two mechanisms conflict?
> > > > > >
> > > > > > In addition to the types listed there, Flink supports
> > reading/writing
> > > > > nullable types. Flink maps nullable types to Avro union(something,
> > > null),
> > > > > where something is the Avro type converted from Flink type.
> > > > > > Is that definitely the right way round? I know we've had multiple
> > > > > conversations about how unions work with Flink
> > > > > >
> > > > > >  This is because the writer schema is expanded, but this could
> not
> > > > > complete if there are circularities.
> > > > > > I understand your meaning but the sentence is awkward.
> > > > > >
> > > > > > The registered schema will be created or if it exists be updated.
> > > > > > same again
> > > > > >
> > > > > > At some stage the lowest Flink level supported by the Kafka
> > connector
> > > > > will contain the additionalProperties methods in code flink.
> > > > > > wording
> > > > > >
> > > > > > There existing Kafka deserialization for the writer schema passes
> > > down
> > > > > the message body to be deserialised.
> > > > > > wording
> > > > > >
> > > > > > @Override
> > > > > > public void deserialize(ConsumerRecord<byte[], byte[]> message,
> > > > > Collector<T> out)
> > > > > >       throws IOException {
> > > > > >       Map<String, Object> additionalPropertiesMap =  new
> > HashMap<>();
> > > > > >       for (Header header : message.additionalProperties()) {
> > > > > >       headersMap.put(header.key(), header.value());
> > > > > >       }
> > > > > >       deserializationSchema.deserialize(message.value(),
> > headersMap,
> > > > > out);
> > > > > > }
> > > > > > This fails to compile at headersMap.
> > > > > >
> > > > > > The input stream and additionalProperties will be sent so the
> > > Apicurio
> > > > > SchemaCoder which will try getting the globalId from the headers,
> > then
> > > 4
> > > > > bytes from the payload then 8 bytes from the payload.
> > > > > > I'm still stuck on apicurio-avro.globalid-placement having a
> > default
> > > > > value of HEADER . Should we try all three, or fail if this config
> > param
> > > > has
> > > > > a wrong value?
> > > > > >
> > > > > > Other considerations
> > > > > > The implementation does not use the Apicurio deser libraries,
> > > > > > Please can we refer to them as SerDes; this is the term used
> within
> > > the
> > > > > documentation that you link to
> > > > > >
> > > > > >
> > > > > > On 2024/03/20 10:09:08 David Radley wrote:
> > > > > > > Hi,
> > > > > > > As per the FLIP process I would like to raise a FLIP, but do
> not
> > > have
> > > > > authority, so have created a google doc for the Flip to introduce a
> > new
> > > > > Apicurio Avro format. The document is
> > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/14LWZPVFQ7F9mryJPdKXb4l32n7B0iWYkcOdEd1xTC7w/edit?usp=sharing
> > > > > > >
> > > > > > > I have prototyped a lot of the content to prove that this
> > approach
> > > is
> > > > > feasible. I look forward to the discussion,
> > > > > > >       Kind regards, David.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Unless otherwise stated above:
> > > > > > >
> > > > > > > IBM United Kingdom Limited
> > > > > > > Registered in England and Wales with number 741598
> > > > > > > Registered office: PO Box 41, North Harbour, Portsmouth, Hants.
> > PO6
> > > > 3AU
> > > > > > >
> > > > > >
> > > > >
> > > >
> > > > Unless otherwise stated above:
> > > >
> > > > IBM United Kingdom Limited
> > > > Registered in England and Wales with number 741598
> > > > Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6
> 3AU
> > > >
> > >
> > > Unless otherwise stated above:
> > >
> > > IBM United Kingdom Limited
> > > Registered in England and Wales with number 741598
> > > Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU
> > >
> >
> > Unless otherwise stated above:
> >
> > IBM United Kingdom Limited
> > Registered in England and Wales with number 741598
> > Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU
> >
>
> Unless otherwise stated above:
>
> IBM United Kingdom Limited
> Registered in England and Wales with number 741598
> Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU
>

Unless otherwise stated above:

IBM United Kingdom Limited
Registered in England and Wales with number 741598
Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU

Unless otherwise stated above:

IBM United Kingdom Limited
Registered in England and Wales with number 741598
Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU

RE: FW: RE: [DISCUSS] FLIP-XXX Apicurio-avro format

Posted by David Radley <da...@uk.ibm.com>.
Fabulous, thanks Martijn 😊

From: Martijn Visser <ma...@apache.org>
Date: Thursday, 2 May 2024 at 10:08
To: dev@flink.apache.org <de...@flink.apache.org>
Subject: [EXTERNAL] Re: FW: RE: [DISCUSS] FLIP-XXX Apicurio-avro format
Done :)

On Thu, May 2, 2024 at 11:01 AM David Radley <da...@uk.ibm.com>
wrote:

> Hi Martijn,
> Thank you very much for looking at this. In response to your feedback; I
> produced a reduced version which is on this link.
>
>
> https://docs.google.com/document/d/1J1E-cE-X2H3-kw4rNjLn71OGPQk_Yl1iGX4-eCHWLgE/edit?usp=sharing
>
> The original version you have copied is a bit out-dated and verbose.
> Please could you replace the Flip with content from the above link,
>                                 Kind regards, David,
>
> From: Martijn Visser <ma...@apache.org>
> Date: Wednesday, 1 May 2024 at 16:31
> To: dev@flink.apache.org <de...@flink.apache.org>
> Subject: [EXTERNAL] Re: FW: RE: [DISCUSS] FLIP-XXX Apicurio-avro format
> Hi David,
>
> I've copied and pasted it into
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-454%3A+New+Apicurio+Avro+format
> ;
> please take a look if it's as expected.
>
> Best regards,
>
> Martijn
>
> On Wed, May 1, 2024 at 3:47 PM David Radley <da...@uk.ibm.com>
> wrote:
>
> > Hi Martijn,
> > Any news?
> >            Kind regards, David.
> >
> >
> > From: David Radley <da...@uk.ibm.com>
> > Date: Monday, 22 April 2024 at 09:48
> > To: dev@flink.apache.org <de...@flink.apache.org>
> > Subject: FW: [EXTERNAL] RE: [DISCUSS] FLIP-XXX Apicurio-avro format
> > Hi Martijn,
> > A gentle nudge, is this ok for you or one of the PMC or committers to
> > create a Flip now?
> >            Kind regards, David.
> >
> > From: David Radley <da...@uk.ibm.com>
> > Date: Monday, 15 April 2024 at 12:29
> > To: dev@flink.apache.org <de...@flink.apache.org>
> > Subject: Re: [EXTERNAL] RE: [DISCUSS] FLIP-XXX Apicurio-avro format
> > Hi Martijn,
> > Thanks for looking at this. I have used the template in a new  Google Doc
> >
> https://docs.google.com/document/d/1J1E-cE-X2H3-kw4rNjLn71OGPQk_Yl1iGX4-eCHWLgE/edit?usp=sharing
> .
> > I have significantly reduced the content in the Flip, in line with what I
> > see as the template and its usage. If this it too much or too little, I
> can
> > amend,
> >
> >     Kind regards, David.
> >
> > From: Martijn Visser <ma...@apache.org>
> > Date: Friday, 12 April 2024 at 18:11
> > To: dev@flink.apache.org <de...@flink.apache.org>
> > Subject: Re: [EXTERNAL] RE: [DISCUSS] FLIP-XXX Apicurio-avro format
> > Hi David,
> >
> > I tried, but the format wasn't as the FLIP template expects, so I ended
> up
> > needing to change the entire formatting and that was just too much work
> to
> > be honest. If you could make sure that especially the headers match with
> > the FLIP template, and that all of the contents from the FLIP template is
> > there, that would make things much easier.
> >
> > Thanks,
> >
> > Martijn
> >
> > On Fri, Apr 12, 2024 at 6:08 PM David Radley <da...@uk.ibm.com>
> > wrote:
> >
> > > Hi,
> > > A gentle nudge. Please could a committer/PMC member raise the Flip for
> > > this,
> > >       Kind regards, David.
> > >
> > >
> > > From: David Radley <da...@uk.ibm.com>
> > > Date: Monday, 8 April 2024 at 09:40
> > > To: dev@flink.apache.org <de...@flink.apache.org>
> > > Subject: [EXTERNAL] RE: [DISCUSS] FLIP-XXX Apicurio-avro format
> > > Hi,
> > > I have posted a Google Doc [0] to the mailing list for a discussion
> > thread
> > > for a Flip proposal to introduce a Apicurio-avro format. The
> discussions
> > > have been resolved, please could a committer/PMC member copy the
> contents
> > > from the Google Doc, and create a FLIP number for this,. as per the
> > process
> > > [1],
> > >       Kind regards, David.
> > > [0]
> > >
> > >
> >
> https://docs.google.com/document/d/14LWZPVFQ7F9mryJPdKXb4l32n7B0iWYkcOdEd1xTC7w/edit?usp=sharing
> > >
> > > [1]
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals#FlinkImprovementProposals-CreateyourOwnFLIP
> > >
> > > From: Jeyhun Karimov <je...@gmail.com>
> > > Date: Friday, 22 March 2024 at 13:05
> > > To: dev@flink.apache.org <de...@flink.apache.org>
> > > Subject: [EXTERNAL] Re: [DISCUSS] FLIP-XXX Apicurio-avro format
> > > Hi David,
> > >
> > > Thanks a lot for clarification.
> > > Sounds good to me.
> > >
> > > Regards,
> > > Jeyhun
> > >
> > > On Fri, Mar 22, 2024 at 10:54 AM David Radley <david_radley@uk.ibm.com
> >
> > > wrote:
> > >
> > > > Hi Jeyhun,
> > > > Thanks for your feedback.
> > > >
> > > > So for outbound messages, the message includes the global ID. We
> > register
> > > > the schema and match on the artifact id. So if the schema then
> evolved,
> > > > adding a new  version, the global ID would still be unique and the
> same
> > > > version would be targeted. If you wanted to change the Flink table
> > > > definition in line with a higher version, then you could do this –
> the
> > > > artifact id would need to match for it to use the same schema and a
> > > higher
> > > > artifact version would need to be provided. I notice that Apicurio
> has
> > > > rules around compatibility that you can configure, I suppose if we
> > > attempt
> > > > to create an artifact that breaks these rules , then the register
> > schema
> > > > will fail and the associated operation should fail (e.g. an insert).
> I
> > > have
> > > > not tried this.
> > > >
> > > >
> > > > For inbound messages, using the global id in the header – this
> targets
> > > one
> > > > version of the schema. I can create different messages on the topic
> > built
> > > > with different schema versions, and I can create different tables in
> > > Flink,
> > > > as long as the reader and writer schemas are compatible as per the
> > > >
> > >
> >
> https://github.com/apache/flink/blob/779459168c46b7b4c600ef52f99a5435f81b9048/flink-formats/flink-avro/src/main/java/org/apache/flink/formats/avro/RegistryAvroDeserializationSchema.java#L109
> > > > Then this should work.
> > > >
> > > > Does this address your question?
> > > >     Kind regards, David.
> > > >
> > > >
> > > > From: Jeyhun Karimov <je...@gmail.com>
> > > > Date: Thursday, 21 March 2024 at 21:06
> > > > To: dev@flink.apache.org <de...@flink.apache.org>
> > > > Subject: [EXTERNAL] Re: [DISCUSS] FLIP-XXX Apicurio-avro format
> > > > Hi David,
> > > >
> > > > Thanks for the FLIP. +1 for it.
> > > > I have a minor comment.
> > > >
> > > > Can you please elaborate more on mechanisms in place to ensure data
> > > > consistency and integrity, particularly in the event of schema
> > conflicts?
> > > > Since each message includes a schema ID for inbound and outbound
> > > messages,
> > > > can you elaborate more on message consistency in the context of
> schema
> > > > evolution?
> > > >
> > > > Regards,
> > > > Jeyhun
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On Wed, Mar 20, 2024 at 4:34 PM David Radley <da...@apache.org>
> > > wrote:
> > > >
> > > > > Thank you very much for your feedback Mark. I have made the changes
> > in
> > > > the
> > > > > latest google document. On reflection I agree with you that the
> > > > > globalIdPlacement format configuration should apply to the
> > > > deserialization
> > > > > as well, so it is declarative. I am also going to have a new
> > > > configuration
> > > > > option to work with content IDs as well as global IDs. In line with
> > the
> > > > > deser Apicurio IdHandler and headerHandlers.
> > > > >
> > > > >  kind regards, David.
> > > > >
> > > > >
> > > > > On 2024/03/20 15:18:37 Mark Nuttall wrote:
> > > > > > +1 to this
> > > > > >
> > > > > > A few small comments:
> > > > > >
> > > > > > Currently, if users have Avro schemas in an Apicurio Registry (an
> > > open
> > > > > source Apache 2 licensed schema registry), then the natural way to
> > work
> > > > > with those Avro flows is to use the schemas in the Apicurio
> > Repository.
> > > > > > 'those Avro flows' ... this is the first reference to flows.
> > > > > >
> > > > > > The new format will use the global Id to look up the Avro schema
> > that
> > > > > the message was written during deserialization.
> > > > > > I get the point, phrasing is awkward. Probably you're more
> > interested
> > > > in
> > > > > content than word polish at this point though.
> > > > > >
> > > > > > The Avro Schema Registry (apicurio-avro) format
> > > > > > The Confluent format is called avro-confluent; this should be
> > > > > avro-apicurio
> > > > > >
> > > > > > How to create tables with Apicurio-avro format
> > > > > > s/Apicurio-avro/avro-apicurio/g
> > > > > >
> > > > > > HEADER – globalId is put in the header
> > > > > > LEGACY– global Id is put in the message as a long
> > > > > > CONFLUENT - globalId is put in the message as an int.
> > > > > > Please could we specify 'four-byte int' and 'eight-byte long' ?
> > > > > >
> > > > > > For a Kafka source the globalId will be looked for in this order:
> > > > > > -     In the header
> > > > > > -     After a magic byte as an int
> > > > > > -     After a magic byte as a long.
> > > > > > but apicurio-avro.globalid-placement has a default value of
> HEADER
> > :
> > > > why
> > > > > do we have a search order as well? Isn't
> > > apicurio-avro.globalid-placement
> > > > > enough? Don't the two mechanisms conflict?
> > > > > >
> > > > > > In addition to the types listed there, Flink supports
> > reading/writing
> > > > > nullable types. Flink maps nullable types to Avro union(something,
> > > null),
> > > > > where something is the Avro type converted from Flink type.
> > > > > > Is that definitely the right way round? I know we've had multiple
> > > > > conversations about how unions work with Flink
> > > > > >
> > > > > >  This is because the writer schema is expanded, but this could
> not
> > > > > complete if there are circularities.
> > > > > > I understand your meaning but the sentence is awkward.
> > > > > >
> > > > > > The registered schema will be created or if it exists be updated.
> > > > > > same again
> > > > > >
> > > > > > At some stage the lowest Flink level supported by the Kafka
> > connector
> > > > > will contain the additionalProperties methods in code flink.
> > > > > > wording
> > > > > >
> > > > > > There existing Kafka deserialization for the writer schema passes
> > > down
> > > > > the message body to be deserialised.
> > > > > > wording
> > > > > >
> > > > > > @Override
> > > > > > public void deserialize(ConsumerRecord<byte[], byte[]> message,
> > > > > Collector<T> out)
> > > > > >       throws IOException {
> > > > > >       Map<String, Object> additionalPropertiesMap =  new
> > HashMap<>();
> > > > > >       for (Header header : message.additionalProperties()) {
> > > > > >       headersMap.put(header.key(), header.value());
> > > > > >       }
> > > > > >       deserializationSchema.deserialize(message.value(),
> > headersMap,
> > > > > out);
> > > > > > }
> > > > > > This fails to compile at headersMap.
> > > > > >
> > > > > > The input stream and additionalProperties will be sent so the
> > > Apicurio
> > > > > SchemaCoder which will try getting the globalId from the headers,
> > then
> > > 4
> > > > > bytes from the payload then 8 bytes from the payload.
> > > > > > I'm still stuck on apicurio-avro.globalid-placement having a
> > default
> > > > > value of HEADER . Should we try all three, or fail if this config
> > param
> > > > has
> > > > > a wrong value?
> > > > > >
> > > > > > Other considerations
> > > > > > The implementation does not use the Apicurio deser libraries,
> > > > > > Please can we refer to them as SerDes; this is the term used
> within
> > > the
> > > > > documentation that you link to
> > > > > >
> > > > > >
> > > > > > On 2024/03/20 10:09:08 David Radley wrote:
> > > > > > > Hi,
> > > > > > > As per the FLIP process I would like to raise a FLIP, but do
> not
> > > have
> > > > > authority, so have created a google doc for the Flip to introduce a
> > new
> > > > > Apicurio Avro format. The document is
> > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/14LWZPVFQ7F9mryJPdKXb4l32n7B0iWYkcOdEd1xTC7w/edit?usp=sharing
> > > > > > >
> > > > > > > I have prototyped a lot of the content to prove that this
> > approach
> > > is
> > > > > feasible. I look forward to the discussion,
> > > > > > >       Kind regards, David.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Unless otherwise stated above:
> > > > > > >
> > > > > > > IBM United Kingdom Limited
> > > > > > > Registered in England and Wales with number 741598
> > > > > > > Registered office: PO Box 41, North Harbour, Portsmouth, Hants.
> > PO6
> > > > 3AU
> > > > > > >
> > > > > >
> > > > >
> > > >
> > > > Unless otherwise stated above:
> > > >
> > > > IBM United Kingdom Limited
> > > > Registered in England and Wales with number 741598
> > > > Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6
> 3AU
> > > >
> > >
> > > Unless otherwise stated above:
> > >
> > > IBM United Kingdom Limited
> > > Registered in England and Wales with number 741598
> > > Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU
> > >
> >
> > Unless otherwise stated above:
> >
> > IBM United Kingdom Limited
> > Registered in England and Wales with number 741598
> > Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU
> >
>
> Unless otherwise stated above:
>
> IBM United Kingdom Limited
> Registered in England and Wales with number 741598
> Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU
>

Unless otherwise stated above:

IBM United Kingdom Limited
Registered in England and Wales with number 741598
Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU

Re: FW: RE: [DISCUSS] FLIP-XXX Apicurio-avro format

Posted by Martijn Visser <ma...@apache.org>.
Done :)

On Thu, May 2, 2024 at 11:01 AM David Radley <da...@uk.ibm.com>
wrote:

> Hi Martijn,
> Thank you very much for looking at this. In response to your feedback; I
> produced a reduced version which is on this link.
>
>
> https://docs.google.com/document/d/1J1E-cE-X2H3-kw4rNjLn71OGPQk_Yl1iGX4-eCHWLgE/edit?usp=sharing
>
> The original version you have copied is a bit out-dated and verbose.
> Please could you replace the Flip with content from the above link,
>                                 Kind regards, David,
>
> From: Martijn Visser <ma...@apache.org>
> Date: Wednesday, 1 May 2024 at 16:31
> To: dev@flink.apache.org <de...@flink.apache.org>
> Subject: [EXTERNAL] Re: FW: RE: [DISCUSS] FLIP-XXX Apicurio-avro format
> Hi David,
>
> I've copied and pasted it into
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-454%3A+New+Apicurio+Avro+format
> ;
> please take a look if it's as expected.
>
> Best regards,
>
> Martijn
>
> On Wed, May 1, 2024 at 3:47 PM David Radley <da...@uk.ibm.com>
> wrote:
>
> > Hi Martijn,
> > Any news?
> >            Kind regards, David.
> >
> >
> > From: David Radley <da...@uk.ibm.com>
> > Date: Monday, 22 April 2024 at 09:48
> > To: dev@flink.apache.org <de...@flink.apache.org>
> > Subject: FW: [EXTERNAL] RE: [DISCUSS] FLIP-XXX Apicurio-avro format
> > Hi Martijn,
> > A gentle nudge, is this ok for you or one of the PMC or committers to
> > create a Flip now?
> >            Kind regards, David.
> >
> > From: David Radley <da...@uk.ibm.com>
> > Date: Monday, 15 April 2024 at 12:29
> > To: dev@flink.apache.org <de...@flink.apache.org>
> > Subject: Re: [EXTERNAL] RE: [DISCUSS] FLIP-XXX Apicurio-avro format
> > Hi Martijn,
> > Thanks for looking at this. I have used the template in a new  Google Doc
> >
> https://docs.google.com/document/d/1J1E-cE-X2H3-kw4rNjLn71OGPQk_Yl1iGX4-eCHWLgE/edit?usp=sharing
> .
> > I have significantly reduced the content in the Flip, in line with what I
> > see as the template and its usage. If this it too much or too little, I
> can
> > amend,
> >
> >     Kind regards, David.
> >
> > From: Martijn Visser <ma...@apache.org>
> > Date: Friday, 12 April 2024 at 18:11
> > To: dev@flink.apache.org <de...@flink.apache.org>
> > Subject: Re: [EXTERNAL] RE: [DISCUSS] FLIP-XXX Apicurio-avro format
> > Hi David,
> >
> > I tried, but the format wasn't as the FLIP template expects, so I ended
> up
> > needing to change the entire formatting and that was just too much work
> to
> > be honest. If you could make sure that especially the headers match with
> > the FLIP template, and that all of the contents from the FLIP template is
> > there, that would make things much easier.
> >
> > Thanks,
> >
> > Martijn
> >
> > On Fri, Apr 12, 2024 at 6:08 PM David Radley <da...@uk.ibm.com>
> > wrote:
> >
> > > Hi,
> > > A gentle nudge. Please could a committer/PMC member raise the Flip for
> > > this,
> > >       Kind regards, David.
> > >
> > >
> > > From: David Radley <da...@uk.ibm.com>
> > > Date: Monday, 8 April 2024 at 09:40
> > > To: dev@flink.apache.org <de...@flink.apache.org>
> > > Subject: [EXTERNAL] RE: [DISCUSS] FLIP-XXX Apicurio-avro format
> > > Hi,
> > > I have posted a Google Doc [0] to the mailing list for a discussion
> > thread
> > > for a Flip proposal to introduce a Apicurio-avro format. The
> discussions
> > > have been resolved, please could a committer/PMC member copy the
> contents
> > > from the Google Doc, and create a FLIP number for this,. as per the
> > process
> > > [1],
> > >       Kind regards, David.
> > > [0]
> > >
> > >
> >
> https://docs.google.com/document/d/14LWZPVFQ7F9mryJPdKXb4l32n7B0iWYkcOdEd1xTC7w/edit?usp=sharing
> > >
> > > [1]
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals#FlinkImprovementProposals-CreateyourOwnFLIP
> > >
> > > From: Jeyhun Karimov <je...@gmail.com>
> > > Date: Friday, 22 March 2024 at 13:05
> > > To: dev@flink.apache.org <de...@flink.apache.org>
> > > Subject: [EXTERNAL] Re: [DISCUSS] FLIP-XXX Apicurio-avro format
> > > Hi David,
> > >
> > > Thanks a lot for clarification.
> > > Sounds good to me.
> > >
> > > Regards,
> > > Jeyhun
> > >
> > > On Fri, Mar 22, 2024 at 10:54 AM David Radley <david_radley@uk.ibm.com
> >
> > > wrote:
> > >
> > > > Hi Jeyhun,
> > > > Thanks for your feedback.
> > > >
> > > > So for outbound messages, the message includes the global ID. We
> > register
> > > > the schema and match on the artifact id. So if the schema then
> evolved,
> > > > adding a new  version, the global ID would still be unique and the
> same
> > > > version would be targeted. If you wanted to change the Flink table
> > > > definition in line with a higher version, then you could do this –
> the
> > > > artifact id would need to match for it to use the same schema and a
> > > higher
> > > > artifact version would need to be provided. I notice that Apicurio
> has
> > > > rules around compatibility that you can configure, I suppose if we
> > > attempt
> > > > to create an artifact that breaks these rules , then the register
> > schema
> > > > will fail and the associated operation should fail (e.g. an insert).
> I
> > > have
> > > > not tried this.
> > > >
> > > >
> > > > For inbound messages, using the global id in the header – this
> targets
> > > one
> > > > version of the schema. I can create different messages on the topic
> > built
> > > > with different schema versions, and I can create different tables in
> > > Flink,
> > > > as long as the reader and writer schemas are compatible as per the
> > > >
> > >
> >
> https://github.com/apache/flink/blob/779459168c46b7b4c600ef52f99a5435f81b9048/flink-formats/flink-avro/src/main/java/org/apache/flink/formats/avro/RegistryAvroDeserializationSchema.java#L109
> > > > Then this should work.
> > > >
> > > > Does this address your question?
> > > >     Kind regards, David.
> > > >
> > > >
> > > > From: Jeyhun Karimov <je...@gmail.com>
> > > > Date: Thursday, 21 March 2024 at 21:06
> > > > To: dev@flink.apache.org <de...@flink.apache.org>
> > > > Subject: [EXTERNAL] Re: [DISCUSS] FLIP-XXX Apicurio-avro format
> > > > Hi David,
> > > >
> > > > Thanks for the FLIP. +1 for it.
> > > > I have a minor comment.
> > > >
> > > > Can you please elaborate more on mechanisms in place to ensure data
> > > > consistency and integrity, particularly in the event of schema
> > conflicts?
> > > > Since each message includes a schema ID for inbound and outbound
> > > messages,
> > > > can you elaborate more on message consistency in the context of
> schema
> > > > evolution?
> > > >
> > > > Regards,
> > > > Jeyhun
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On Wed, Mar 20, 2024 at 4:34 PM David Radley <da...@apache.org>
> > > wrote:
> > > >
> > > > > Thank you very much for your feedback Mark. I have made the changes
> > in
> > > > the
> > > > > latest google document. On reflection I agree with you that the
> > > > > globalIdPlacement format configuration should apply to the
> > > > deserialization
> > > > > as well, so it is declarative. I am also going to have a new
> > > > configuration
> > > > > option to work with content IDs as well as global IDs. In line with
> > the
> > > > > deser Apicurio IdHandler and headerHandlers.
> > > > >
> > > > >  kind regards, David.
> > > > >
> > > > >
> > > > > On 2024/03/20 15:18:37 Mark Nuttall wrote:
> > > > > > +1 to this
> > > > > >
> > > > > > A few small comments:
> > > > > >
> > > > > > Currently, if users have Avro schemas in an Apicurio Registry (an
> > > open
> > > > > source Apache 2 licensed schema registry), then the natural way to
> > work
> > > > > with those Avro flows is to use the schemas in the Apicurio
> > Repository.
> > > > > > 'those Avro flows' ... this is the first reference to flows.
> > > > > >
> > > > > > The new format will use the global Id to look up the Avro schema
> > that
> > > > > the message was written during deserialization.
> > > > > > I get the point, phrasing is awkward. Probably you're more
> > interested
> > > > in
> > > > > content than word polish at this point though.
> > > > > >
> > > > > > The Avro Schema Registry (apicurio-avro) format
> > > > > > The Confluent format is called avro-confluent; this should be
> > > > > avro-apicurio
> > > > > >
> > > > > > How to create tables with Apicurio-avro format
> > > > > > s/Apicurio-avro/avro-apicurio/g
> > > > > >
> > > > > > HEADER – globalId is put in the header
> > > > > > LEGACY– global Id is put in the message as a long
> > > > > > CONFLUENT - globalId is put in the message as an int.
> > > > > > Please could we specify 'four-byte int' and 'eight-byte long' ?
> > > > > >
> > > > > > For a Kafka source the globalId will be looked for in this order:
> > > > > > -     In the header
> > > > > > -     After a magic byte as an int
> > > > > > -     After a magic byte as a long.
> > > > > > but apicurio-avro.globalid-placement has a default value of
> HEADER
> > :
> > > > why
> > > > > do we have a search order as well? Isn't
> > > apicurio-avro.globalid-placement
> > > > > enough? Don't the two mechanisms conflict?
> > > > > >
> > > > > > In addition to the types listed there, Flink supports
> > reading/writing
> > > > > nullable types. Flink maps nullable types to Avro union(something,
> > > null),
> > > > > where something is the Avro type converted from Flink type.
> > > > > > Is that definitely the right way round? I know we've had multiple
> > > > > conversations about how unions work with Flink
> > > > > >
> > > > > >  This is because the writer schema is expanded, but this could
> not
> > > > > complete if there are circularities.
> > > > > > I understand your meaning but the sentence is awkward.
> > > > > >
> > > > > > The registered schema will be created or if it exists be updated.
> > > > > > same again
> > > > > >
> > > > > > At some stage the lowest Flink level supported by the Kafka
> > connector
> > > > > will contain the additionalProperties methods in code flink.
> > > > > > wording
> > > > > >
> > > > > > There existing Kafka deserialization for the writer schema passes
> > > down
> > > > > the message body to be deserialised.
> > > > > > wording
> > > > > >
> > > > > > @Override
> > > > > > public void deserialize(ConsumerRecord<byte[], byte[]> message,
> > > > > Collector<T> out)
> > > > > >       throws IOException {
> > > > > >       Map<String, Object> additionalPropertiesMap =  new
> > HashMap<>();
> > > > > >       for (Header header : message.additionalProperties()) {
> > > > > >       headersMap.put(header.key(), header.value());
> > > > > >       }
> > > > > >       deserializationSchema.deserialize(message.value(),
> > headersMap,
> > > > > out);
> > > > > > }
> > > > > > This fails to compile at headersMap.
> > > > > >
> > > > > > The input stream and additionalProperties will be sent so the
> > > Apicurio
> > > > > SchemaCoder which will try getting the globalId from the headers,
> > then
> > > 4
> > > > > bytes from the payload then 8 bytes from the payload.
> > > > > > I'm still stuck on apicurio-avro.globalid-placement having a
> > default
> > > > > value of HEADER . Should we try all three, or fail if this config
> > param
> > > > has
> > > > > a wrong value?
> > > > > >
> > > > > > Other considerations
> > > > > > The implementation does not use the Apicurio deser libraries,
> > > > > > Please can we refer to them as SerDes; this is the term used
> within
> > > the
> > > > > documentation that you link to
> > > > > >
> > > > > >
> > > > > > On 2024/03/20 10:09:08 David Radley wrote:
> > > > > > > Hi,
> > > > > > > As per the FLIP process I would like to raise a FLIP, but do
> not
> > > have
> > > > > authority, so have created a google doc for the Flip to introduce a
> > new
> > > > > Apicurio Avro format. The document is
> > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/14LWZPVFQ7F9mryJPdKXb4l32n7B0iWYkcOdEd1xTC7w/edit?usp=sharing
> > > > > > >
> > > > > > > I have prototyped a lot of the content to prove that this
> > approach
> > > is
> > > > > feasible. I look forward to the discussion,
> > > > > > >       Kind regards, David.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Unless otherwise stated above:
> > > > > > >
> > > > > > > IBM United Kingdom Limited
> > > > > > > Registered in England and Wales with number 741598
> > > > > > > Registered office: PO Box 41, North Harbour, Portsmouth, Hants.
> > PO6
> > > > 3AU
> > > > > > >
> > > > > >
> > > > >
> > > >
> > > > Unless otherwise stated above:
> > > >
> > > > IBM United Kingdom Limited
> > > > Registered in England and Wales with number 741598
> > > > Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6
> 3AU
> > > >
> > >
> > > Unless otherwise stated above:
> > >
> > > IBM United Kingdom Limited
> > > Registered in England and Wales with number 741598
> > > Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU
> > >
> >
> > Unless otherwise stated above:
> >
> > IBM United Kingdom Limited
> > Registered in England and Wales with number 741598
> > Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU
> >
>
> Unless otherwise stated above:
>
> IBM United Kingdom Limited
> Registered in England and Wales with number 741598
> Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU
>

RE: FW: RE: [DISCUSS] FLIP-XXX Apicurio-avro format

Posted by David Radley <da...@uk.ibm.com>.
Hi Martijn,
Thank you very much for looking at this. In response to your feedback; I produced a reduced version which is on this link.

https://docs.google.com/document/d/1J1E-cE-X2H3-kw4rNjLn71OGPQk_Yl1iGX4-eCHWLgE/edit?usp=sharing

The original version you have copied is a bit out-dated and verbose. Please could you replace the Flip with content from the above link,
                                Kind regards, David,

From: Martijn Visser <ma...@apache.org>
Date: Wednesday, 1 May 2024 at 16:31
To: dev@flink.apache.org <de...@flink.apache.org>
Subject: [EXTERNAL] Re: FW: RE: [DISCUSS] FLIP-XXX Apicurio-avro format
Hi David,

I've copied and pasted it into
https://cwiki.apache.org/confluence/display/FLINK/FLIP-454%3A+New+Apicurio+Avro+format;
please take a look if it's as expected.

Best regards,

Martijn

On Wed, May 1, 2024 at 3:47 PM David Radley <da...@uk.ibm.com> wrote:

> Hi Martijn,
> Any news?
>            Kind regards, David.
>
>
> From: David Radley <da...@uk.ibm.com>
> Date: Monday, 22 April 2024 at 09:48
> To: dev@flink.apache.org <de...@flink.apache.org>
> Subject: FW: [EXTERNAL] RE: [DISCUSS] FLIP-XXX Apicurio-avro format
> Hi Martijn,
> A gentle nudge, is this ok for you or one of the PMC or committers to
> create a Flip now?
>            Kind regards, David.
>
> From: David Radley <da...@uk.ibm.com>
> Date: Monday, 15 April 2024 at 12:29
> To: dev@flink.apache.org <de...@flink.apache.org>
> Subject: Re: [EXTERNAL] RE: [DISCUSS] FLIP-XXX Apicurio-avro format
> Hi Martijn,
> Thanks for looking at this. I have used the template in a new  Google Doc
> https://docs.google.com/document/d/1J1E-cE-X2H3-kw4rNjLn71OGPQk_Yl1iGX4-eCHWLgE/edit?usp=sharing.
> I have significantly reduced the content in the Flip, in line with what I
> see as the template and its usage. If this it too much or too little, I can
> amend,
>
>     Kind regards, David.
>
> From: Martijn Visser <ma...@apache.org>
> Date: Friday, 12 April 2024 at 18:11
> To: dev@flink.apache.org <de...@flink.apache.org>
> Subject: Re: [EXTERNAL] RE: [DISCUSS] FLIP-XXX Apicurio-avro format
> Hi David,
>
> I tried, but the format wasn't as the FLIP template expects, so I ended up
> needing to change the entire formatting and that was just too much work to
> be honest. If you could make sure that especially the headers match with
> the FLIP template, and that all of the contents from the FLIP template is
> there, that would make things much easier.
>
> Thanks,
>
> Martijn
>
> On Fri, Apr 12, 2024 at 6:08 PM David Radley <da...@uk.ibm.com>
> wrote:
>
> > Hi,
> > A gentle nudge. Please could a committer/PMC member raise the Flip for
> > this,
> >       Kind regards, David.
> >
> >
> > From: David Radley <da...@uk.ibm.com>
> > Date: Monday, 8 April 2024 at 09:40
> > To: dev@flink.apache.org <de...@flink.apache.org>
> > Subject: [EXTERNAL] RE: [DISCUSS] FLIP-XXX Apicurio-avro format
> > Hi,
> > I have posted a Google Doc [0] to the mailing list for a discussion
> thread
> > for a Flip proposal to introduce a Apicurio-avro format. The discussions
> > have been resolved, please could a committer/PMC member copy the contents
> > from the Google Doc, and create a FLIP number for this,. as per the
> process
> > [1],
> >       Kind regards, David.
> > [0]
> >
> >
> https://docs.google.com/document/d/14LWZPVFQ7F9mryJPdKXb4l32n7B0iWYkcOdEd1xTC7w/edit?usp=sharing
> >
> > [1]
> >
> >
> https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals#FlinkImprovementProposals-CreateyourOwnFLIP
> >
> > From: Jeyhun Karimov <je...@gmail.com>
> > Date: Friday, 22 March 2024 at 13:05
> > To: dev@flink.apache.org <de...@flink.apache.org>
> > Subject: [EXTERNAL] Re: [DISCUSS] FLIP-XXX Apicurio-avro format
> > Hi David,
> >
> > Thanks a lot for clarification.
> > Sounds good to me.
> >
> > Regards,
> > Jeyhun
> >
> > On Fri, Mar 22, 2024 at 10:54 AM David Radley <da...@uk.ibm.com>
> > wrote:
> >
> > > Hi Jeyhun,
> > > Thanks for your feedback.
> > >
> > > So for outbound messages, the message includes the global ID. We
> register
> > > the schema and match on the artifact id. So if the schema then evolved,
> > > adding a new  version, the global ID would still be unique and the same
> > > version would be targeted. If you wanted to change the Flink table
> > > definition in line with a higher version, then you could do this – the
> > > artifact id would need to match for it to use the same schema and a
> > higher
> > > artifact version would need to be provided. I notice that Apicurio has
> > > rules around compatibility that you can configure, I suppose if we
> > attempt
> > > to create an artifact that breaks these rules , then the register
> schema
> > > will fail and the associated operation should fail (e.g. an insert). I
> > have
> > > not tried this.
> > >
> > >
> > > For inbound messages, using the global id in the header – this targets
> > one
> > > version of the schema. I can create different messages on the topic
> built
> > > with different schema versions, and I can create different tables in
> > Flink,
> > > as long as the reader and writer schemas are compatible as per the
> > >
> >
> https://github.com/apache/flink/blob/779459168c46b7b4c600ef52f99a5435f81b9048/flink-formats/flink-avro/src/main/java/org/apache/flink/formats/avro/RegistryAvroDeserializationSchema.java#L109
> > > Then this should work.
> > >
> > > Does this address your question?
> > >     Kind regards, David.
> > >
> > >
> > > From: Jeyhun Karimov <je...@gmail.com>
> > > Date: Thursday, 21 March 2024 at 21:06
> > > To: dev@flink.apache.org <de...@flink.apache.org>
> > > Subject: [EXTERNAL] Re: [DISCUSS] FLIP-XXX Apicurio-avro format
> > > Hi David,
> > >
> > > Thanks for the FLIP. +1 for it.
> > > I have a minor comment.
> > >
> > > Can you please elaborate more on mechanisms in place to ensure data
> > > consistency and integrity, particularly in the event of schema
> conflicts?
> > > Since each message includes a schema ID for inbound and outbound
> > messages,
> > > can you elaborate more on message consistency in the context of schema
> > > evolution?
> > >
> > > Regards,
> > > Jeyhun
> > >
> > >
> > >
> > >
> > >
> > > On Wed, Mar 20, 2024 at 4:34 PM David Radley <da...@apache.org>
> > wrote:
> > >
> > > > Thank you very much for your feedback Mark. I have made the changes
> in
> > > the
> > > > latest google document. On reflection I agree with you that the
> > > > globalIdPlacement format configuration should apply to the
> > > deserialization
> > > > as well, so it is declarative. I am also going to have a new
> > > configuration
> > > > option to work with content IDs as well as global IDs. In line with
> the
> > > > deser Apicurio IdHandler and headerHandlers.
> > > >
> > > >  kind regards, David.
> > > >
> > > >
> > > > On 2024/03/20 15:18:37 Mark Nuttall wrote:
> > > > > +1 to this
> > > > >
> > > > > A few small comments:
> > > > >
> > > > > Currently, if users have Avro schemas in an Apicurio Registry (an
> > open
> > > > source Apache 2 licensed schema registry), then the natural way to
> work
> > > > with those Avro flows is to use the schemas in the Apicurio
> Repository.
> > > > > 'those Avro flows' ... this is the first reference to flows.
> > > > >
> > > > > The new format will use the global Id to look up the Avro schema
> that
> > > > the message was written during deserialization.
> > > > > I get the point, phrasing is awkward. Probably you're more
> interested
> > > in
> > > > content than word polish at this point though.
> > > > >
> > > > > The Avro Schema Registry (apicurio-avro) format
> > > > > The Confluent format is called avro-confluent; this should be
> > > > avro-apicurio
> > > > >
> > > > > How to create tables with Apicurio-avro format
> > > > > s/Apicurio-avro/avro-apicurio/g
> > > > >
> > > > > HEADER – globalId is put in the header
> > > > > LEGACY– global Id is put in the message as a long
> > > > > CONFLUENT - globalId is put in the message as an int.
> > > > > Please could we specify 'four-byte int' and 'eight-byte long' ?
> > > > >
> > > > > For a Kafka source the globalId will be looked for in this order:
> > > > > -     In the header
> > > > > -     After a magic byte as an int
> > > > > -     After a magic byte as a long.
> > > > > but apicurio-avro.globalid-placement has a default value of HEADER
> :
> > > why
> > > > do we have a search order as well? Isn't
> > apicurio-avro.globalid-placement
> > > > enough? Don't the two mechanisms conflict?
> > > > >
> > > > > In addition to the types listed there, Flink supports
> reading/writing
> > > > nullable types. Flink maps nullable types to Avro union(something,
> > null),
> > > > where something is the Avro type converted from Flink type.
> > > > > Is that definitely the right way round? I know we've had multiple
> > > > conversations about how unions work with Flink
> > > > >
> > > > >  This is because the writer schema is expanded, but this could not
> > > > complete if there are circularities.
> > > > > I understand your meaning but the sentence is awkward.
> > > > >
> > > > > The registered schema will be created or if it exists be updated.
> > > > > same again
> > > > >
> > > > > At some stage the lowest Flink level supported by the Kafka
> connector
> > > > will contain the additionalProperties methods in code flink.
> > > > > wording
> > > > >
> > > > > There existing Kafka deserialization for the writer schema passes
> > down
> > > > the message body to be deserialised.
> > > > > wording
> > > > >
> > > > > @Override
> > > > > public void deserialize(ConsumerRecord<byte[], byte[]> message,
> > > > Collector<T> out)
> > > > >       throws IOException {
> > > > >       Map<String, Object> additionalPropertiesMap =  new
> HashMap<>();
> > > > >       for (Header header : message.additionalProperties()) {
> > > > >       headersMap.put(header.key(), header.value());
> > > > >       }
> > > > >       deserializationSchema.deserialize(message.value(),
> headersMap,
> > > > out);
> > > > > }
> > > > > This fails to compile at headersMap.
> > > > >
> > > > > The input stream and additionalProperties will be sent so the
> > Apicurio
> > > > SchemaCoder which will try getting the globalId from the headers,
> then
> > 4
> > > > bytes from the payload then 8 bytes from the payload.
> > > > > I'm still stuck on apicurio-avro.globalid-placement having a
> default
> > > > value of HEADER . Should we try all three, or fail if this config
> param
> > > has
> > > > a wrong value?
> > > > >
> > > > > Other considerations
> > > > > The implementation does not use the Apicurio deser libraries,
> > > > > Please can we refer to them as SerDes; this is the term used within
> > the
> > > > documentation that you link to
> > > > >
> > > > >
> > > > > On 2024/03/20 10:09:08 David Radley wrote:
> > > > > > Hi,
> > > > > > As per the FLIP process I would like to raise a FLIP, but do not
> > have
> > > > authority, so have created a google doc for the Flip to introduce a
> new
> > > > Apicurio Avro format. The document is
> > > >
> > >
> >
> https://docs.google.com/document/d/14LWZPVFQ7F9mryJPdKXb4l32n7B0iWYkcOdEd1xTC7w/edit?usp=sharing
> > > > > >
> > > > > > I have prototyped a lot of the content to prove that this
> approach
> > is
> > > > feasible. I look forward to the discussion,
> > > > > >       Kind regards, David.
> > > > > >
> > > > > >
> > > > > >
> > > > > > Unless otherwise stated above:
> > > > > >
> > > > > > IBM United Kingdom Limited
> > > > > > Registered in England and Wales with number 741598
> > > > > > Registered office: PO Box 41, North Harbour, Portsmouth, Hants.
> PO6
> > > 3AU
> > > > > >
> > > > >
> > > >
> > >
> > > Unless otherwise stated above:
> > >
> > > IBM United Kingdom Limited
> > > Registered in England and Wales with number 741598
> > > Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU
> > >
> >
> > Unless otherwise stated above:
> >
> > IBM United Kingdom Limited
> > Registered in England and Wales with number 741598
> > Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU
> >
>
> Unless otherwise stated above:
>
> IBM United Kingdom Limited
> Registered in England and Wales with number 741598
> Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU
>

Unless otherwise stated above:

IBM United Kingdom Limited
Registered in England and Wales with number 741598
Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU

Re: FW: RE: [DISCUSS] FLIP-XXX Apicurio-avro format

Posted by Martijn Visser <ma...@apache.org>.
Hi David,

I've copied and pasted it into
https://cwiki.apache.org/confluence/display/FLINK/FLIP-454%3A+New+Apicurio+Avro+format;
please take a look if it's as expected.

Best regards,

Martijn

On Wed, May 1, 2024 at 3:47 PM David Radley <da...@uk.ibm.com> wrote:

> Hi Martijn,
> Any news?
>            Kind regards, David.
>
>
> From: David Radley <da...@uk.ibm.com>
> Date: Monday, 22 April 2024 at 09:48
> To: dev@flink.apache.org <de...@flink.apache.org>
> Subject: FW: [EXTERNAL] RE: [DISCUSS] FLIP-XXX Apicurio-avro format
> Hi Martijn,
> A gentle nudge, is this ok for you or one of the PMC or committers to
> create a Flip now?
>            Kind regards, David.
>
> From: David Radley <da...@uk.ibm.com>
> Date: Monday, 15 April 2024 at 12:29
> To: dev@flink.apache.org <de...@flink.apache.org>
> Subject: Re: [EXTERNAL] RE: [DISCUSS] FLIP-XXX Apicurio-avro format
> Hi Martijn,
> Thanks for looking at this. I have used the template in a new  Google Doc
> https://docs.google.com/document/d/1J1E-cE-X2H3-kw4rNjLn71OGPQk_Yl1iGX4-eCHWLgE/edit?usp=sharing.
> I have significantly reduced the content in the Flip, in line with what I
> see as the template and its usage. If this it too much or too little, I can
> amend,
>
>     Kind regards, David.
>
> From: Martijn Visser <ma...@apache.org>
> Date: Friday, 12 April 2024 at 18:11
> To: dev@flink.apache.org <de...@flink.apache.org>
> Subject: Re: [EXTERNAL] RE: [DISCUSS] FLIP-XXX Apicurio-avro format
> Hi David,
>
> I tried, but the format wasn't as the FLIP template expects, so I ended up
> needing to change the entire formatting and that was just too much work to
> be honest. If you could make sure that especially the headers match with
> the FLIP template, and that all of the contents from the FLIP template is
> there, that would make things much easier.
>
> Thanks,
>
> Martijn
>
> On Fri, Apr 12, 2024 at 6:08 PM David Radley <da...@uk.ibm.com>
> wrote:
>
> > Hi,
> > A gentle nudge. Please could a committer/PMC member raise the Flip for
> > this,
> >       Kind regards, David.
> >
> >
> > From: David Radley <da...@uk.ibm.com>
> > Date: Monday, 8 April 2024 at 09:40
> > To: dev@flink.apache.org <de...@flink.apache.org>
> > Subject: [EXTERNAL] RE: [DISCUSS] FLIP-XXX Apicurio-avro format
> > Hi,
> > I have posted a Google Doc [0] to the mailing list for a discussion
> thread
> > for a Flip proposal to introduce a Apicurio-avro format. The discussions
> > have been resolved, please could a committer/PMC member copy the contents
> > from the Google Doc, and create a FLIP number for this,. as per the
> process
> > [1],
> >       Kind regards, David.
> > [0]
> >
> >
> https://docs.google.com/document/d/14LWZPVFQ7F9mryJPdKXb4l32n7B0iWYkcOdEd1xTC7w/edit?usp=sharing
> >
> > [1]
> >
> >
> https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals#FlinkImprovementProposals-CreateyourOwnFLIP
> >
> > From: Jeyhun Karimov <je...@gmail.com>
> > Date: Friday, 22 March 2024 at 13:05
> > To: dev@flink.apache.org <de...@flink.apache.org>
> > Subject: [EXTERNAL] Re: [DISCUSS] FLIP-XXX Apicurio-avro format
> > Hi David,
> >
> > Thanks a lot for clarification.
> > Sounds good to me.
> >
> > Regards,
> > Jeyhun
> >
> > On Fri, Mar 22, 2024 at 10:54 AM David Radley <da...@uk.ibm.com>
> > wrote:
> >
> > > Hi Jeyhun,
> > > Thanks for your feedback.
> > >
> > > So for outbound messages, the message includes the global ID. We
> register
> > > the schema and match on the artifact id. So if the schema then evolved,
> > > adding a new  version, the global ID would still be unique and the same
> > > version would be targeted. If you wanted to change the Flink table
> > > definition in line with a higher version, then you could do this – the
> > > artifact id would need to match for it to use the same schema and a
> > higher
> > > artifact version would need to be provided. I notice that Apicurio has
> > > rules around compatibility that you can configure, I suppose if we
> > attempt
> > > to create an artifact that breaks these rules , then the register
> schema
> > > will fail and the associated operation should fail (e.g. an insert). I
> > have
> > > not tried this.
> > >
> > >
> > > For inbound messages, using the global id in the header – this targets
> > one
> > > version of the schema. I can create different messages on the topic
> built
> > > with different schema versions, and I can create different tables in
> > Flink,
> > > as long as the reader and writer schemas are compatible as per the
> > >
> >
> https://github.com/apache/flink/blob/779459168c46b7b4c600ef52f99a5435f81b9048/flink-formats/flink-avro/src/main/java/org/apache/flink/formats/avro/RegistryAvroDeserializationSchema.java#L109
> > > Then this should work.
> > >
> > > Does this address your question?
> > >     Kind regards, David.
> > >
> > >
> > > From: Jeyhun Karimov <je...@gmail.com>
> > > Date: Thursday, 21 March 2024 at 21:06
> > > To: dev@flink.apache.org <de...@flink.apache.org>
> > > Subject: [EXTERNAL] Re: [DISCUSS] FLIP-XXX Apicurio-avro format
> > > Hi David,
> > >
> > > Thanks for the FLIP. +1 for it.
> > > I have a minor comment.
> > >
> > > Can you please elaborate more on mechanisms in place to ensure data
> > > consistency and integrity, particularly in the event of schema
> conflicts?
> > > Since each message includes a schema ID for inbound and outbound
> > messages,
> > > can you elaborate more on message consistency in the context of schema
> > > evolution?
> > >
> > > Regards,
> > > Jeyhun
> > >
> > >
> > >
> > >
> > >
> > > On Wed, Mar 20, 2024 at 4:34 PM David Radley <da...@apache.org>
> > wrote:
> > >
> > > > Thank you very much for your feedback Mark. I have made the changes
> in
> > > the
> > > > latest google document. On reflection I agree with you that the
> > > > globalIdPlacement format configuration should apply to the
> > > deserialization
> > > > as well, so it is declarative. I am also going to have a new
> > > configuration
> > > > option to work with content IDs as well as global IDs. In line with
> the
> > > > deser Apicurio IdHandler and headerHandlers.
> > > >
> > > >  kind regards, David.
> > > >
> > > >
> > > > On 2024/03/20 15:18:37 Mark Nuttall wrote:
> > > > > +1 to this
> > > > >
> > > > > A few small comments:
> > > > >
> > > > > Currently, if users have Avro schemas in an Apicurio Registry (an
> > open
> > > > source Apache 2 licensed schema registry), then the natural way to
> work
> > > > with those Avro flows is to use the schemas in the Apicurio
> Repository.
> > > > > 'those Avro flows' ... this is the first reference to flows.
> > > > >
> > > > > The new format will use the global Id to look up the Avro schema
> that
> > > > the message was written during deserialization.
> > > > > I get the point, phrasing is awkward. Probably you're more
> interested
> > > in
> > > > content than word polish at this point though.
> > > > >
> > > > > The Avro Schema Registry (apicurio-avro) format
> > > > > The Confluent format is called avro-confluent; this should be
> > > > avro-apicurio
> > > > >
> > > > > How to create tables with Apicurio-avro format
> > > > > s/Apicurio-avro/avro-apicurio/g
> > > > >
> > > > > HEADER – globalId is put in the header
> > > > > LEGACY– global Id is put in the message as a long
> > > > > CONFLUENT - globalId is put in the message as an int.
> > > > > Please could we specify 'four-byte int' and 'eight-byte long' ?
> > > > >
> > > > > For a Kafka source the globalId will be looked for in this order:
> > > > > -     In the header
> > > > > -     After a magic byte as an int
> > > > > -     After a magic byte as a long.
> > > > > but apicurio-avro.globalid-placement has a default value of HEADER
> :
> > > why
> > > > do we have a search order as well? Isn't
> > apicurio-avro.globalid-placement
> > > > enough? Don't the two mechanisms conflict?
> > > > >
> > > > > In addition to the types listed there, Flink supports
> reading/writing
> > > > nullable types. Flink maps nullable types to Avro union(something,
> > null),
> > > > where something is the Avro type converted from Flink type.
> > > > > Is that definitely the right way round? I know we've had multiple
> > > > conversations about how unions work with Flink
> > > > >
> > > > >  This is because the writer schema is expanded, but this could not
> > > > complete if there are circularities.
> > > > > I understand your meaning but the sentence is awkward.
> > > > >
> > > > > The registered schema will be created or if it exists be updated.
> > > > > same again
> > > > >
> > > > > At some stage the lowest Flink level supported by the Kafka
> connector
> > > > will contain the additionalProperties methods in code flink.
> > > > > wording
> > > > >
> > > > > There existing Kafka deserialization for the writer schema passes
> > down
> > > > the message body to be deserialised.
> > > > > wording
> > > > >
> > > > > @Override
> > > > > public void deserialize(ConsumerRecord<byte[], byte[]> message,
> > > > Collector<T> out)
> > > > >       throws IOException {
> > > > >       Map<String, Object> additionalPropertiesMap =  new
> HashMap<>();
> > > > >       for (Header header : message.additionalProperties()) {
> > > > >       headersMap.put(header.key(), header.value());
> > > > >       }
> > > > >       deserializationSchema.deserialize(message.value(),
> headersMap,
> > > > out);
> > > > > }
> > > > > This fails to compile at headersMap.
> > > > >
> > > > > The input stream and additionalProperties will be sent so the
> > Apicurio
> > > > SchemaCoder which will try getting the globalId from the headers,
> then
> > 4
> > > > bytes from the payload then 8 bytes from the payload.
> > > > > I'm still stuck on apicurio-avro.globalid-placement having a
> default
> > > > value of HEADER . Should we try all three, or fail if this config
> param
> > > has
> > > > a wrong value?
> > > > >
> > > > > Other considerations
> > > > > The implementation does not use the Apicurio deser libraries,
> > > > > Please can we refer to them as SerDes; this is the term used within
> > the
> > > > documentation that you link to
> > > > >
> > > > >
> > > > > On 2024/03/20 10:09:08 David Radley wrote:
> > > > > > Hi,
> > > > > > As per the FLIP process I would like to raise a FLIP, but do not
> > have
> > > > authority, so have created a google doc for the Flip to introduce a
> new
> > > > Apicurio Avro format. The document is
> > > >
> > >
> >
> https://docs.google.com/document/d/14LWZPVFQ7F9mryJPdKXb4l32n7B0iWYkcOdEd1xTC7w/edit?usp=sharing
> > > > > >
> > > > > > I have prototyped a lot of the content to prove that this
> approach
> > is
> > > > feasible. I look forward to the discussion,
> > > > > >       Kind regards, David.
> > > > > >
> > > > > >
> > > > > >
> > > > > > Unless otherwise stated above:
> > > > > >
> > > > > > IBM United Kingdom Limited
> > > > > > Registered in England and Wales with number 741598
> > > > > > Registered office: PO Box 41, North Harbour, Portsmouth, Hants.
> PO6
> > > 3AU
> > > > > >
> > > > >
> > > >
> > >
> > > Unless otherwise stated above:
> > >
> > > IBM United Kingdom Limited
> > > Registered in England and Wales with number 741598
> > > Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU
> > >
> >
> > Unless otherwise stated above:
> >
> > IBM United Kingdom Limited
> > Registered in England and Wales with number 741598
> > Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU
> >
>
> Unless otherwise stated above:
>
> IBM United Kingdom Limited
> Registered in England and Wales with number 741598
> Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU
>