Logical type for vector

[Philipp Fischbeck via dev <de...@avro.apache.org>]

Hi folks,

In the Parquet and Iceberg communities, we are having ongoing discussions
about introducing new data types for vector data, i.e., fixed-size lists of
numeric values that typically represent vector embeddings for machine
learning contexts.

In Parquet [1], we will likely introduce a new logical type FINITE_VECTOR
with a property "num_elements" that annotates a 3-level LIST of numeric
values.

In Iceberg [2], we want to introduce a similar vector type. For Parquet
data files, Iceberg tables will use this newly introduced Parquet logical
type. We are now discussing how best to approach Avro files [3], and wanted
to get feedback from the Avro community.
One way would be to use the "array" type, and annotate it with a
logicalType "vector" and a "num_elements" (or similar field). Would you be
interested in standardizing such a vector annotation in Avro? Or are there
alternatives that make more sense?

Best,
Philipp

[1] https://github.com/apache/parquet-format/pull/624
[2] https://github.com/apache/iceberg/pull/18416
[3] https://iceberg.apache.org/spec/#avro