[DISCUSS] Numeric Vector type: physical representation (outcome of focused call)

[Rok Mihevc <ro...@gmail.com>]

Hi all,

As proposed during the general community call we held a focused call today
to discuss the proposal for adding a logical vector type to parquet. This
was called because it seems:

* Vectors (as in vector DB) storage needs are different to what we need for
fixed size list applications we've been disucssing so far [0]. This is
driven also by Iceberg's need for Vector type [1]
* We don't yet have a consensus of what physical representation of Vector
type should be.


Notes from the call can be found here [2]. Short summary:

We have converged on two options that could work, but have not yet decided
which one to propose:

- Option A - FLBA as physical storage (FLBA(num_elements * element_width)).
Vector logical type would provide an enum specifying the element type, and
the element would be non-nullable. The element type enum could be
independent of the parquet type system, making evolution easier. Downside
is loss of element-level encodings available today, though they can be
added later. Upside is simplicity and ease of evolution.

- Option C1 - LIST with numeric parquet type elements as physical storage.
New float types (e.g. BF16) can be added in parquet proper as needed.
Downside is we need to add new types to parquet, DL/RL overhead is likely
not a concern. Upside is we have encoding support now.

Regarding NaN/Inf limitations, the consensus was that these should be
handled by statistics. This leads to a need for a new inf_count statustics.

It would be great to discuss the merits of these two proposals here on the
mailing list for maximum visibility and participation.
We will likely continue to discuss this on tomorrow's community call too.


[0] https://lists.apache.org/thread/xot5f3ghhtc82n1bf0wdl9zqwlrzqks3
[1]
https://docs.google.com/document/d/1zA8PskNDFKXXJpzWQr25CFoX_aIZHBNHbkxk0t3PteE/edit?tab=t.0#heading=h.j5pn3fef87b2
[2]
https://docs.google.com/document/d/1rPTjEJTwN97UfHvNlvdriX9XBQsW8PGwIHMXHvy-XlU/edit?tab=t.0


Best,
Rok

Re: [DISCUSS] Numeric Vector type: physical representation (outcome of focused call)

[Rok Mihevc <ro...@gmail.com>]

Thanks for that thought Will!

> So is Parquet just growing a vector column type for the fixed-size
> embeddings used in nearest-neighbour search to keep up with mainstream
> databases, or is it jumping to support the broader set of ML tensor data
to
> accommodate all the random things an ML engineer might want to jam into a
> Parquet file today?  I guess that is the question?

Could we have both.

The original intent of the design proposal was to cover a general
fixed size list that could accommodate various multidimensional array
applications (e.g. [1][2]). It seems we would likely want a breaking
change for this. This is an effort I want to continue working on.

However, there is a lot of immediate interest in resolving the limited
vector
use case, and that without breaking changes.

I proposed specifying the limited vector first to decouple the discussion
and design of the general fixed size list from the limited vector type
requirements. However this might not be a strong enough motivation to
introduce a new logical type.

[1]
https://arrow.apache.org/docs/format/CanonicalExtensions.html#fixed-shape-tensor
[2]
https://arrow.apache.org/docs/format/CanonicalExtensions.html#variable-shape-tensor
[3] https://github.com/apache/parquet-format/pull/451

Best,
Rok

Re: [DISCUSS] Numeric Vector type: physical representation (outcome of focused call)

[Rok Mihevc <ro...@gmail.com>]

Thanks for that thought Will!

> So is Parquet just growing a vector column type for the fixed-size
> embeddings used in nearest-neighbour search to keep up with mainstream
> databases, or is it jumping to support the broader set of ML tensor data
to
> accommodate all the random things an ML engineer might want to jam into a
> Parquet file today?  I guess that is the question?

Could we have both.

The original intent of the design proposal was to cover a general
fixed size list that could accommodate various multidimensional array
applications (e.g. [1][2]). It seems we would likely want a breaking
change for this. This is an effort I want to continue working on.

However, there is a lot of immediate interest in resolving the limited
vector
use case, and that without breaking changes.

I proposed specifying the limited vector first to decouple the discussion
and design of the general fixed size list from the limited vector type
requirements. However this might not be a strong enough motivation to
introduce a new logical type.

[1]
https://arrow.apache.org/docs/format/CanonicalExtensions.html#fixed-shape-tensor
[2]
https://arrow.apache.org/docs/format/CanonicalExtensions.html#variable-shape-tensor
[3] https://github.com/apache/parquet-format/pull/451

Best,
Rok

Re: [DISCUSS] Numeric Vector type: physical representation (outcome of focused call)

[Will Edwards via dev <de...@parquet.apache.org>]

First, regarding NaNs in vectors, that's a good question Russell :D: it's
common enough in ML dev to get them from your models on overflow, etc., and
if you're not thinking about handling them, it's easy enough for them to
end up going unguarded into output?  Production models ought to be
clamping and filtering but I just searched the internet and found a few
threads complaining that OpenAI's text-embedding intermittently returns
NaNs instead of valid embeddings.  Weird but seems true.

More generally, beyond fixed-size embeddings, there are various use-cases
for NaNs in feature vectors and Infs in probability outputs?  Probably
more?  There are also sparse vectors, FP16/8/4/1, scaling etc (pgvector
recently added support for most of them too).

Secondly, stepping back, some things have changed as this conversation has
gone on and on :)   For example, early on there were performance concerns,
but it's been established that Parquet's LIST has no significant
performance penalty for float embeddings.

To recap that, a column might be declared with an arbitrary number of
dimensions and a writer could insert a jagged shape in there filled with
nulls.  However, if all rows actually have the same shape, this is
trivially detectable at the data page level. This allows for readers that
care to have fast paths or very cheap error checks.  Likewise, NaNs and
Infs can be trivially spotted at decode time, or else a reader could just
let them propagate through the math and do one-cycle bitfield isnan check
on the result.

So we've moved the conversation from physical storage to logical types and
have been talking about a 1D fixed-size vector logical type to represent
embeddings, with constraints on nulls and infs and nans.

Howeevr, I fear that a VECTOR(FLOAT, 768) logical type doesn't deliver what
it appears to promise. It states "fixed-length, non-null, finite" - all
things the reader can already detect per-page for free. And it omits the
one factor that truly matters for correctness: whether two columns share
the same embedding space. That's application-level knowledge (which model,
which training run) that belongs in metadata, not the type system.

So strong opinion weakly held:

* No performance problem. Regular data compresses the levels to
near-nothing, and the regularity is trivially detectable from the encoded
streams. Hardwood demonstrated this concretely.

* No correctness problem. Nulls are visible at decode time. NaNs/Infs are
either filtered early or propagate harmlessly through math and caught by a
bitmask check on the result.  And if we had a logical type defining fixed
sizes etc, a defensive reader would check the values anyway to ensure they
conform- it would be a brave engine that forwent the check that is so cheap
anyway.

* No expressiveness problem. Arbitrary-dimensional tensors already work as
nested lists. Dense vs ragged is a property of the data, not the schema,
and readers adapt per-page.

So perhaps the type annotation solves the wrong problem` Structural
compatibility (same length, same element type) doesn't imply semantic
compatibility (same embedding space). The type gives false confidence while
the real constraint remains the caller's responsibility.

So its only a software problem not a Parquet problem if an engine that
supports reading Parquet doesn't offer good-performance distance functions
for LIST<FLOAT> embeddings.  I don't think a type annotation enables
functionality in a way that the user couldn't ask the engine to do the same
thing without the type annotation.

Where Parquet is behind is in the various schemes for smaller floats,
smaller ints, scaling, sparsity etc?  Currently, all that is likely done in
different ways by different writers using FLBA in custom ways.  It's likely
that only who wrote the file can basically understand and use those fields?

If the logical type was unified and could describe a VECTOR<BF16>[256] (or,
well, pretty much everything the superset of nummy and pytorch and arrow
etc supports :D) and engines would understand how to unpack and use that,
then that would IMO be what moves this into enabling new usecases and
engine interoperability.

Sorry for the wall of text :)

On Thu, 24 Sept 2026 at 20:12, Russell Spitzer <ru...@gmail.com>
wrote:

> I think something I haven't seen yet is the clear use-case for non-finite
> values within a vector type. There has been a lot of theoretical discussion
> here about use cases but what toolings currently produce these vectors and
> how do they function in downstream engines? I'd like to limit introducing
> types that have a wide range of values which are simply are failure modes
> of different flavors
>
> I also don't find any issue with the attitude of folks in this discussion,
> it seems reasonable that people are trying to work together to get things
> done in the format for the practical use cases they know about.
>
> On Thu, Sep 24, 2026 at 8:49 AM Antoine Pitrou <an...@python.org> wrote:
>
> > Le 24/09/2026 à 08:48, Will Edwards via dev a écrit :
> > >
> > > So is Parquet just growing a vector column type for the fixed-size
> > > embeddings used in nearest-neighbour search to keep up with mainstream
> > > databases, or is it jumping to support the broader set of ML tensor
> data
> > to
> > > accommodate all the random things an ML engineer might want to jam
> into a
> > > Parquet file today?  I guess that is the question?
> >
> > Yes, that is a question and that was the heart of my position as well:
> > """
> > Well, Parquet is not a single-purpose format.
> >
> > Vector stores can choose whatever policy fits their usage, but we're
> > talking about a general-purpose file format that aims to be broadly
> > applicable.
> > """
> >
> > I didn't see any objection to this point.
> >
> > I also didn't see any explanation why allowing non-finite values would
> > actually hamper database usage. There are precedents where we don't
> > impose limitations to data types which may be further limited in
> > databases (for example timestamps).
> >
> > I also didn't see any consideration for the more logical solution of
> > adding metadata to tell whether a numeric column may contain non-finite
> > values (regardless of whether it's in a vector or not).
> >
> > And in the end I find that some people here are having a very cavalier
> > attitude towards discussion and reaching consensus.
> >
> > Regards
> >
> > Antoine.
> >
> >
> >
>

Re: [DISCUSS] Numeric Vector type: physical representation (outcome of focused call)

[Julien Le Dem via dev <de...@parquet.apache.org>]

I don't have a strong opinion on the VECTOR type requirement, however I'd
like to make the following meta point:
It is easy to stall while working on building consensus to add new types
and we need to have a bias to action to make sur we progress.
A good way to make this easier is to narrow down the focus of proposed
changes.
In this case, there is a clear need for a specific use case now while we're
also discussing possible additional use cases.
I think that the proposal to narrow this type down to something more
specific (like a "FINITE_VECTOR") is a great way to enable moving forward
with the need we have clarified today.
This opens up the possibility to create more specific vector types in the
future without confusion. This approach puts the other vector type use
cases on hold while unblocking the current one.
The possible downside? In the future, we might have more vector types than
if we had waited more to build an all encompassing type. However to me
there is a very clear trade off: It is better to add more features
incrementally than staying stuck too long on defining the perfect solution
(correctness will always take precedence though. By not perfect I do not
mean possibly incorrect).
We might have some duplication in the future but that's better than moving
too slowly or completely stall.
Best
Julien

On Mon, Nov 2, 2026 at 2:37 PM Rok Mihevc <ro...@gmail.com> wrote:

> Thank you for writing up the alternative proposal Jan!
>
> > I still believe that FINITE_VECTOR, as proposed in PR #624
> > <https://github.com/apache/parquet-format/pull/624>, is not the right
> > abstraction for a general-purpose format because it lacks composability.
>
> FINITE_VECTOR is proposed as a narrow semantic type, aimed at a
> specific use case. We clearly also want a general-purpose fixed size list
> type (idealy a physical storage optimisation too) and FINITE_VECTOR
> does not preclude that, but we should pursue it as a separate effort.
>
> > *Finite floats:* Give floating-point types an optional isFinite property.
>
> While this would work - is anyone asking for finite elements other than
> this FINITE_VECTOR proposal?
>
> > Higher order tensors. You can just have list<list<finite float, 64>, 64>>
> for example.
>
> I don't think Parquet should model tensors (multidimensional arrays) this
> way.
> A common way to model tensors is a contiguous fixed size list of values
> plus metadata for shape and strides (dimension order) [1][2][3]. This
> matches
> how numeric libraries lay out tensors in memory, and strides allow for
> optimizations on data layout. If we model tensors as nested lists we get
> redundant repetition levels and likely still want to introduce strides to
>  represent
> non-trivial memory layouts.
>
> [1]
>
> https://arrow.apache.org/docs/format/CanonicalExtensions.html#fixed-shape-tensor
> [2] https://pytorch.org/blog/tensor-memory-format-matters/
> [3] https://blog.ezyang.com/2019/05/pytorch-internals/
>
> Best,
> Rok
>
> On Mon, Nov 2, 2026 at 8:43 PM Jan Finis <jp...@gmail.com> wrote:
>
> > I still believe that FINITE_VECTOR, as proposed in PR #624
> > <https://github.com/apache/parquet-format/pull/624>, is not the right
> > abstraction for a general-purpose format because it lacks composability.
> >
> > I've drafted an alternative proposal (PR #633)
> > <https://github.com/apache/parquet-format/pull/633> to demonstrate how
> we
> > could achieve the same functionality in a more composable way. This is by
> > no means ready to merge; it's primarily intended to illustrate the
> > approach.
> >
> > The idea is simple: introduce two orthogonal building blocks:
> >
> >    -
> >
> >    *Fixed-size lists:* Give lists an optional elementCount.
> >    -
> >
> >    *Finite floats:* Give floating-point types an optional isFinite
> > property.
> >
> > These two additions would allow us to express FINITE_VECTOR, but also
> > enable a much broader range of useful types:
> >
> >    -
> >
> >    Fixed-size lists of *any* element type, not just those supported by
> >    FINITE_VECTOR.
> >    -
> >
> >    Finite floats in any context, whether standalone, in fixed- or
> >    variable-size lists, in maps, or elsewhere.
> >    -
> >
> >    Higher order tensors. You can just have list<list<finite float, 64>,
> >    64>> for example.
> >
> > As a bonus, fixed-size lists would remain readable by legacy readers,
> which
> > would simply interpret them as variable-size lists.
> > As finiteness is applied only to floats, we also don't get the semantic
> > weirdness where FINITE_VECTOR can also annotate a vector of bools or
> ints,
> > which are trivially always finite.
> >
> > My main concern with FINITE_VECTOR is that it's highly specific to
> > embedding vectors. While embeddings are certainly a hot topic today, I
> > think baking such feature-specific abstractions into a general-purpose
> > format intended to remain relevant for decades is the wrong trade-off.
> > Different systems have different requirements, and those requirements
> will
> > evolve.
> >
> > I'd much rather see us build a small set of *orthogonal, composable
> types*
> > from which specialized types like FINITE_VECTOR can be constructed,
> rather
> > than introducing those specialized types directly into the format.
> >
> >
> >
> > Am Mo., 2. Nov. 2026 um 20:23 Uhr schrieb Jan Finis <jp...@gmail.com>:
> >
> > > > Yes, I expect different functions in engines. I think pgvector uses
> > more
> > > > specific functions for vectors, like `vector_dims` and
> `vector_norms`.
> > As
> > > > far as I'm aware, these aren't available for arrays without an
> explicit
> > > > cast.
> > >
> > > pg_vector is IMHO just one example that cannot be generalized. The
> reason
> > > (I suspect) that it uses its own type is that PostgreSQL's array type
> is
> > > somewhat
> > > overly bloated. It is not fixed length and it even supports multiple
> > > dimensions, leading
> > > to an overly verbose binary format. Had PostgreSQL a fixed-length array
> > > type that
> > > was just represented as the values in binary, I think that pgvector
> would
> > > not have
> > > chosen an own type.
> > >
> > > In contrast, I would expect most engines that already have a
> fixed-length
> > > array type
> > > to just use that for vectors.
> > >
> > > Am Mo., 2. Nov. 2026 um 15:03 Uhr schrieb Rok Mihevc <
> > rok.mihevc@gmail.com
> > > >:
> > >
> > >> > Could the proposal include a compatibility section addressing
> > >> unannotated
> > >> list data before we vote?
> > >>
> > >> Will, would a paragraph like this [1] be what you have in mind?
> > >>
> > >> On the topic of the vote - I would like to call it later today based
> on
> > >> parquet-format [2].
> > >>
> > >> [1]
> > >>
> > >>
> >
> https://github.com/apache/parquet-format/pull/624/changes/8ad956d543e3074e1d9c8d4faa194199e06a8be4
> > >> [2] https://github.com/apache/parquet-format/pull/624
> > >>
> > >> Best,
> > >> Rok
> > >>
> > >> On Mon, Nov 2, 2026 at 12:42 PM Will Edwards via dev <
> > >> dev@parquet.apache.org>
> > >> wrote:
> > >>
> > >> > Thanks Ryan, Dan.
> > >> >
> > >> > My questions about what the type enables, and how you think Iceberg
> > >> tooling
> > >> > would use it, stem from the backward-compatibility concerns I've
> > raised
> > >> > earlier in the thread. I was trying to weigh the cost against the
> > >> benefit.
> > >> > I worry that this annotation will fragment and orphan a lot of data:
> > >> > perfectly good vectors that become second-class just because they
> > >> weren't
> > >> > written with the annotation.
> > >> >
> > >> > There is an enormous amount of embedding data in Parquet today. Many
> > >> many
> > >> > lottabytes of it :) Some of it is packed in bespoke formats, but a
> > very
> > >> > large share is plain, unannotated lists of floats or doubles. In
> > >> addition
> > >> > to all this existing data, many pipelines will keep writing new data
> > >> this
> > >> > way too, because they are unaware of the new annotation or can't be
> > >> > changed.
> > >> >
> > >> > Ryan, you said compatibility isn't a concern because this is a new
> > >> type. I
> > >> > think that assumes the data is new data written by an Iceberg
> writer.
> > >> But
> > >> > many Iceberg tables are built from files written by other engines.
> The
> > >> > standard implementations support this directly (add_files, migrate,
> > name
> > >> > mapping), and the popular public embedding datasets are a typical
> > >> source.
> > >> > If Iceberg requires vector columns to carry the annotation, none of
> > >> those
> > >> > files can back a vector column without being rewritten, even though
> > the
> > >> > bytes would be identical.
> > >> >
> > >> > It makes sense for Iceberg writers to check for inf/nan/null when
> they
> > >> > write a vector column. But since many files come from other writers,
> > >> robust
> > >> > readers should check correctness anyway, as they do with UTF-8 for
> > >> strings.
> > >> >
> > >> > So my position remains that this would be better done entirely in
> > >> Iceberg
> > >> > rather than pushed down to Parquet. Iceberg ought to accept
> > unannotated
> > >> > LIST<FLOAT>/LIST<DOUBLE> files for vector columns, including via
> > >> add_files
> > >> > and a metadata-only list → vector promotion.
> > >> >
> > >> > Could the proposal include a compatibility section addressing
> > >> unannotated
> > >> > list data before we vote?
> > >> >
> > >> > Will
> > >> >
> > >> > On Sat, 31 Oct 2026 at 00:00, Ryan Blue via dev <
> > dev@parquet.apache.org
> > >> >
> > >> > wrote:
> > >> >
> > >> > > > 1. do you expect that the backing files that are parquet MUST
> use
> > >> the
> > >> > new
> > >> > > logical vector type?
> > >> > >
> > >> > > I expect that the Iceberg spec will require writers to produce the
> > >> > > annotation.
> > >> > >
> > >> > > > 2. do you expect the software accessing  an iceberg table . . .
> to
> > >> > offer
> > >> > > different functions on vector columns compared to naked lists?
> > >> > >
> > >> > > Yes, I expect different functions in engines. I think pgvector
> uses
> > >> more
> > >> > > specific functions for vectors, like `vector_dims` and
> > >> `vector_norms`. As
> > >> > > far as I'm aware, these aren't available for arrays without an
> > >> explicit
> > >> > > cast.
> > >> > >
> > >> > > > Will a nearest neighbour search on a naked list of doubles be
> > >> > supported,
> > >> > > for example?
> > >> > >
> > >> > > This is an engine decision, but a list like this would not be
> > exposed
> > >> as
> > >> > a
> > >> > > vector. You would most likely cast it to a vector in order to call
> > >> vector
> > >> > > functions.
> > >> > >
> > >> > > > 3. And will non-parquet backing files like avro and orc be
> > supported
> > >> > for
> > >> > > the
> > >> > > vector columns?
> > >> > >
> > >> > > All Iceberg types can be stored in supported file formats.
> > >> > >
> > >> > > It seems like the direction for this line of questioning is
> > basically
> > >> the
> > >> > > same as before: Can we just treat list<float> as a synonym of
> > vector?
> > >> > >
> > >> > > And again, I think this is the wrong question. We have users right
> > now
> > >> > that
> > >> > > struggle with errors that are found at read time, like the
> > dimensions
> > >> of
> > >> > > their vectors not matching. Finding this out at read time is a
> > problem
> > >> > and
> > >> > > we want to shift validation to write time so we can catch errors
> > >> earlier
> > >> > > and have guarantees that the data is usable. As I mentioned, one
> of
> > >> the
> > >> > > ways we have customers using data is sharing it with systems like
> > >> > pgvector
> > >> > > that also have these requirements. Just because you can think of a
> > >> vector
> > >> > > as a very special list of floats doesn't mean that write time
> > >> guarantees
> > >> > > are unreasonable or unnecessary.
> > >> > >
> > >> > > Ryan
> > >> > >
> > >> > >
> > >> > > On Thu, Oct 29, 2026 at 12:57 PM Will Edwards via dev <
> > >> > > dev@parquet.apache.org> wrote:
> > >> > >
> > >> > > > Thx for replying.
> > >> > > >
> > >> > > > I keep trying to explain that there are no performance
> advantages.
> > >> But
> > >> > > > anyway :)
> > >> > > >
> > >> > > > I understand you are envisaging a vector column type (with the
> > fixed
> > >> > > length
> > >> > > > nan inf constraints you’ve been detailing) being added to
> iceberg?
> > >> > Then
> > >> > > > the following choices are interesting:
> > >> > > >
> > >> > > > 1. do you expect that the backing files that are parquet MUST
> use
> > >> the
> > >> > new
> > >> > > > logical vector type?
> > >> > > >
> > >> > > > 2. do you expect the software accessing  an iceberg table -
> > >> sparksql,
> > >> > for
> > >> > > > example - to offer different functions on vector columns
> compared
> > to
> > >> > > naked
> > >> > > > lists?  Will a nearest neighbour search on a naked list of
> doubles
> > >> be
> > >> > > > supported, for example?
> > >> > > >
> > >> > > > 3. And will non-parquet backing files like avro and orc be
> > supported
> > >> > for
> > >> > > > the vector columns?
> > >> > > >
> > >> > > > thx
> > >> > > >
> > >> > > > Will
> > >> > > >
> > >> > > > On Sat, 24 Oct 2026 at 02:08, Ryan Blue <bl...@apache.org>
> wrote:
> > >> > > >
> > >> > > > > > I'm asking for a workload that can't be done on LIST<FLOAT>
> > now
> > >> and
> > >> > > > would
> > >> > > > work with VECTOR. Jan asked the same question, and I don't think
> > >> either
> > >> > > of
> > >> > > > us has seen one yet.
> > >> > > >
> > >> > > > I think this is the wrong question to ask, but I'm glad that
> > you're
> > >> > > asking
> > >> > > > directly. Clearly, a variable-length list is a superset of a
> > >> > fixed-length
> > >> > > > list. And a list of floats is a superset of a list of finite
> > >> floats. A
> > >> > > list
> > >> > > > of optional elements is a superset of a list of required
> > elements. I
> > >> > > don't
> > >> > > > think there's a use case that _can't_ be done.
> > >> > > >
> > >> > > > Is there a use case that can't be done with binary in place of
> > >> string?
> > >> > > And
> > >> > > > why don't we use JSON or Variant for everything? Just because we
> > can
> > >> > use
> > >> > > a
> > >> > > > wider type doesn't mean that we don't benefit from communicating
> > >> > > guarantees
> > >> > > > like "this must be UTF-8".
> > >> > > >
> > >> > > > Types introduce guarantees that we can use to get better
> > performance
> > >> > and
> > >> > > to
> > >> > > > either prevent errors or raise them closer to where data is
> > >> produced.
> > >> > > >
> > >> > > > For vectors, the guarantee that elements are finite gives us
> > better
> > >> > > > performance because we don't need to check each element. This
> also
> > >> > moves
> > >> > > > the error closer to where data is produced because writers are
> > >> required
> > >> > > to
> > >> > > > verify the guarantees about each element and will detect bad
> > values
> > >> at
> > >> > > > write time rather than at read time.
> > >> > > >
> > >> > > > Ryan
> > >> > > >
> > >> > >
> > >> >
> > >>
> > >
> >
>

Re: [DISCUSS] Numeric Vector type: physical representation (outcome of focused call)

[Rok Mihevc <ro...@gmail.com>]

Thank you for writing up the alternative proposal Jan!

> I still believe that FINITE_VECTOR, as proposed in PR #624
> <https://github.com/apache/parquet-format/pull/624>, is not the right
> abstraction for a general-purpose format because it lacks composability.

FINITE_VECTOR is proposed as a narrow semantic type, aimed at a
specific use case. We clearly also want a general-purpose fixed size list
type (idealy a physical storage optimisation too) and FINITE_VECTOR
does not preclude that, but we should pursue it as a separate effort.

> *Finite floats:* Give floating-point types an optional isFinite property.

While this would work - is anyone asking for finite elements other than
this FINITE_VECTOR proposal?

> Higher order tensors. You can just have list<list<finite float, 64>, 64>>
for example.

I don't think Parquet should model tensors (multidimensional arrays) this
way.
A common way to model tensors is a contiguous fixed size list of values
plus metadata for shape and strides (dimension order) [1][2][3]. This
matches
how numeric libraries lay out tensors in memory, and strides allow for
optimizations on data layout. If we model tensors as nested lists we get
redundant repetition levels and likely still want to introduce strides to
 represent
non-trivial memory layouts.

[1]
https://arrow.apache.org/docs/format/CanonicalExtensions.html#fixed-shape-tensor
[2] https://pytorch.org/blog/tensor-memory-format-matters/
[3] https://blog.ezyang.com/2019/05/pytorch-internals/

Best,
Rok

On Mon, Nov 2, 2026 at 8:43 PM Jan Finis <jp...@gmail.com> wrote:

> I still believe that FINITE_VECTOR, as proposed in PR #624
> <https://github.com/apache/parquet-format/pull/624>, is not the right
> abstraction for a general-purpose format because it lacks composability.
>
> I've drafted an alternative proposal (PR #633)
> <https://github.com/apache/parquet-format/pull/633> to demonstrate how we
> could achieve the same functionality in a more composable way. This is by
> no means ready to merge; it's primarily intended to illustrate the
> approach.
>
> The idea is simple: introduce two orthogonal building blocks:
>
>    -
>
>    *Fixed-size lists:* Give lists an optional elementCount.
>    -
>
>    *Finite floats:* Give floating-point types an optional isFinite
> property.
>
> These two additions would allow us to express FINITE_VECTOR, but also
> enable a much broader range of useful types:
>
>    -
>
>    Fixed-size lists of *any* element type, not just those supported by
>    FINITE_VECTOR.
>    -
>
>    Finite floats in any context, whether standalone, in fixed- or
>    variable-size lists, in maps, or elsewhere.
>    -
>
>    Higher order tensors. You can just have list<list<finite float, 64>,
>    64>> for example.
>
> As a bonus, fixed-size lists would remain readable by legacy readers, which
> would simply interpret them as variable-size lists.
> As finiteness is applied only to floats, we also don't get the semantic
> weirdness where FINITE_VECTOR can also annotate a vector of bools or ints,
> which are trivially always finite.
>
> My main concern with FINITE_VECTOR is that it's highly specific to
> embedding vectors. While embeddings are certainly a hot topic today, I
> think baking such feature-specific abstractions into a general-purpose
> format intended to remain relevant for decades is the wrong trade-off.
> Different systems have different requirements, and those requirements will
> evolve.
>
> I'd much rather see us build a small set of *orthogonal, composable types*
> from which specialized types like FINITE_VECTOR can be constructed, rather
> than introducing those specialized types directly into the format.
>
>
>
> Am Mo., 2. Nov. 2026 um 20:23 Uhr schrieb Jan Finis <jp...@gmail.com>:
>
> > > Yes, I expect different functions in engines. I think pgvector uses
> more
> > > specific functions for vectors, like `vector_dims` and `vector_norms`.
> As
> > > far as I'm aware, these aren't available for arrays without an explicit
> > > cast.
> >
> > pg_vector is IMHO just one example that cannot be generalized. The reason
> > (I suspect) that it uses its own type is that PostgreSQL's array type is
> > somewhat
> > overly bloated. It is not fixed length and it even supports multiple
> > dimensions, leading
> > to an overly verbose binary format. Had PostgreSQL a fixed-length array
> > type that
> > was just represented as the values in binary, I think that pgvector would
> > not have
> > chosen an own type.
> >
> > In contrast, I would expect most engines that already have a fixed-length
> > array type
> > to just use that for vectors.
> >
> > Am Mo., 2. Nov. 2026 um 15:03 Uhr schrieb Rok Mihevc <
> rok.mihevc@gmail.com
> > >:
> >
> >> > Could the proposal include a compatibility section addressing
> >> unannotated
> >> list data before we vote?
> >>
> >> Will, would a paragraph like this [1] be what you have in mind?
> >>
> >> On the topic of the vote - I would like to call it later today based on
> >> parquet-format [2].
> >>
> >> [1]
> >>
> >>
> https://github.com/apache/parquet-format/pull/624/changes/8ad956d543e3074e1d9c8d4faa194199e06a8be4
> >> [2] https://github.com/apache/parquet-format/pull/624
> >>
> >> Best,
> >> Rok
> >>
> >> On Mon, Nov 2, 2026 at 12:42 PM Will Edwards via dev <
> >> dev@parquet.apache.org>
> >> wrote:
> >>
> >> > Thanks Ryan, Dan.
> >> >
> >> > My questions about what the type enables, and how you think Iceberg
> >> tooling
> >> > would use it, stem from the backward-compatibility concerns I've
> raised
> >> > earlier in the thread. I was trying to weigh the cost against the
> >> benefit.
> >> > I worry that this annotation will fragment and orphan a lot of data:
> >> > perfectly good vectors that become second-class just because they
> >> weren't
> >> > written with the annotation.
> >> >
> >> > There is an enormous amount of embedding data in Parquet today. Many
> >> many
> >> > lottabytes of it :) Some of it is packed in bespoke formats, but a
> very
> >> > large share is plain, unannotated lists of floats or doubles. In
> >> addition
> >> > to all this existing data, many pipelines will keep writing new data
> >> this
> >> > way too, because they are unaware of the new annotation or can't be
> >> > changed.
> >> >
> >> > Ryan, you said compatibility isn't a concern because this is a new
> >> type. I
> >> > think that assumes the data is new data written by an Iceberg writer.
> >> But
> >> > many Iceberg tables are built from files written by other engines. The
> >> > standard implementations support this directly (add_files, migrate,
> name
> >> > mapping), and the popular public embedding datasets are a typical
> >> source.
> >> > If Iceberg requires vector columns to carry the annotation, none of
> >> those
> >> > files can back a vector column without being rewritten, even though
> the
> >> > bytes would be identical.
> >> >
> >> > It makes sense for Iceberg writers to check for inf/nan/null when they
> >> > write a vector column. But since many files come from other writers,
> >> robust
> >> > readers should check correctness anyway, as they do with UTF-8 for
> >> strings.
> >> >
> >> > So my position remains that this would be better done entirely in
> >> Iceberg
> >> > rather than pushed down to Parquet. Iceberg ought to accept
> unannotated
> >> > LIST<FLOAT>/LIST<DOUBLE> files for vector columns, including via
> >> add_files
> >> > and a metadata-only list → vector promotion.
> >> >
> >> > Could the proposal include a compatibility section addressing
> >> unannotated
> >> > list data before we vote?
> >> >
> >> > Will
> >> >
> >> > On Sat, 31 Oct 2026 at 00:00, Ryan Blue via dev <
> dev@parquet.apache.org
> >> >
> >> > wrote:
> >> >
> >> > > > 1. do you expect that the backing files that are parquet MUST use
> >> the
> >> > new
> >> > > logical vector type?
> >> > >
> >> > > I expect that the Iceberg spec will require writers to produce the
> >> > > annotation.
> >> > >
> >> > > > 2. do you expect the software accessing  an iceberg table . . . to
> >> > offer
> >> > > different functions on vector columns compared to naked lists?
> >> > >
> >> > > Yes, I expect different functions in engines. I think pgvector uses
> >> more
> >> > > specific functions for vectors, like `vector_dims` and
> >> `vector_norms`. As
> >> > > far as I'm aware, these aren't available for arrays without an
> >> explicit
> >> > > cast.
> >> > >
> >> > > > Will a nearest neighbour search on a naked list of doubles be
> >> > supported,
> >> > > for example?
> >> > >
> >> > > This is an engine decision, but a list like this would not be
> exposed
> >> as
> >> > a
> >> > > vector. You would most likely cast it to a vector in order to call
> >> vector
> >> > > functions.
> >> > >
> >> > > > 3. And will non-parquet backing files like avro and orc be
> supported
> >> > for
> >> > > the
> >> > > vector columns?
> >> > >
> >> > > All Iceberg types can be stored in supported file formats.
> >> > >
> >> > > It seems like the direction for this line of questioning is
> basically
> >> the
> >> > > same as before: Can we just treat list<float> as a synonym of
> vector?
> >> > >
> >> > > And again, I think this is the wrong question. We have users right
> now
> >> > that
> >> > > struggle with errors that are found at read time, like the
> dimensions
> >> of
> >> > > their vectors not matching. Finding this out at read time is a
> problem
> >> > and
> >> > > we want to shift validation to write time so we can catch errors
> >> earlier
> >> > > and have guarantees that the data is usable. As I mentioned, one of
> >> the
> >> > > ways we have customers using data is sharing it with systems like
> >> > pgvector
> >> > > that also have these requirements. Just because you can think of a
> >> vector
> >> > > as a very special list of floats doesn't mean that write time
> >> guarantees
> >> > > are unreasonable or unnecessary.
> >> > >
> >> > > Ryan
> >> > >
> >> > >
> >> > > On Thu, Oct 29, 2026 at 12:57 PM Will Edwards via dev <
> >> > > dev@parquet.apache.org> wrote:
> >> > >
> >> > > > Thx for replying.
> >> > > >
> >> > > > I keep trying to explain that there are no performance advantages.
> >> But
> >> > > > anyway :)
> >> > > >
> >> > > > I understand you are envisaging a vector column type (with the
> fixed
> >> > > length
> >> > > > nan inf constraints you’ve been detailing) being added to iceberg?
> >> > Then
> >> > > > the following choices are interesting:
> >> > > >
> >> > > > 1. do you expect that the backing files that are parquet MUST use
> >> the
> >> > new
> >> > > > logical vector type?
> >> > > >
> >> > > > 2. do you expect the software accessing  an iceberg table -
> >> sparksql,
> >> > for
> >> > > > example - to offer different functions on vector columns compared
> to
> >> > > naked
> >> > > > lists?  Will a nearest neighbour search on a naked list of doubles
> >> be
> >> > > > supported, for example?
> >> > > >
> >> > > > 3. And will non-parquet backing files like avro and orc be
> supported
> >> > for
> >> > > > the vector columns?
> >> > > >
> >> > > > thx
> >> > > >
> >> > > > Will
> >> > > >
> >> > > > On Sat, 24 Oct 2026 at 02:08, Ryan Blue <bl...@apache.org> wrote:
> >> > > >
> >> > > > > > I'm asking for a workload that can't be done on LIST<FLOAT>
> now
> >> and
> >> > > > would
> >> > > > work with VECTOR. Jan asked the same question, and I don't think
> >> either
> >> > > of
> >> > > > us has seen one yet.
> >> > > >
> >> > > > I think this is the wrong question to ask, but I'm glad that
> you're
> >> > > asking
> >> > > > directly. Clearly, a variable-length list is a superset of a
> >> > fixed-length
> >> > > > list. And a list of floats is a superset of a list of finite
> >> floats. A
> >> > > list
> >> > > > of optional elements is a superset of a list of required
> elements. I
> >> > > don't
> >> > > > think there's a use case that _can't_ be done.
> >> > > >
> >> > > > Is there a use case that can't be done with binary in place of
> >> string?
> >> > > And
> >> > > > why don't we use JSON or Variant for everything? Just because we
> can
> >> > use
> >> > > a
> >> > > > wider type doesn't mean that we don't benefit from communicating
> >> > > guarantees
> >> > > > like "this must be UTF-8".
> >> > > >
> >> > > > Types introduce guarantees that we can use to get better
> performance
> >> > and
> >> > > to
> >> > > > either prevent errors or raise them closer to where data is
> >> produced.
> >> > > >
> >> > > > For vectors, the guarantee that elements are finite gives us
> better
> >> > > > performance because we don't need to check each element. This also
> >> > moves
> >> > > > the error closer to where data is produced because writers are
> >> required
> >> > > to
> >> > > > verify the guarantees about each element and will detect bad
> values
> >> at
> >> > > > write time rather than at read time.
> >> > > >
> >> > > > Ryan
> >> > > >
> >> > >
> >> >
> >>
> >
>

Re: [DISCUSS] Numeric Vector type: physical representation (outcome of focused call)

[Jan Finis <jp...@gmail.com>]

I still believe that FINITE_VECTOR, as proposed in PR #624
<https://github.com/apache/parquet-format/pull/624>, is not the right
abstraction for a general-purpose format because it lacks composability.

I've drafted an alternative proposal (PR #633)
<https://github.com/apache/parquet-format/pull/633> to demonstrate how we
could achieve the same functionality in a more composable way. This is by
no means ready to merge; it's primarily intended to illustrate the approach.

The idea is simple: introduce two orthogonal building blocks:

   -

   *Fixed-size lists:* Give lists an optional elementCount.
   -

   *Finite floats:* Give floating-point types an optional isFinite property.

These two additions would allow us to express FINITE_VECTOR, but also
enable a much broader range of useful types:

   -

   Fixed-size lists of *any* element type, not just those supported by
   FINITE_VECTOR.
   -

   Finite floats in any context, whether standalone, in fixed- or
   variable-size lists, in maps, or elsewhere.
   -

   Higher order tensors. You can just have list<list<finite float, 64>,
   64>> for example.

As a bonus, fixed-size lists would remain readable by legacy readers, which
would simply interpret them as variable-size lists.
As finiteness is applied only to floats, we also don't get the semantic
weirdness where FINITE_VECTOR can also annotate a vector of bools or ints,
which are trivially always finite.

My main concern with FINITE_VECTOR is that it's highly specific to
embedding vectors. While embeddings are certainly a hot topic today, I
think baking such feature-specific abstractions into a general-purpose
format intended to remain relevant for decades is the wrong trade-off.
Different systems have different requirements, and those requirements will
evolve.

I'd much rather see us build a small set of *orthogonal, composable types*
from which specialized types like FINITE_VECTOR can be constructed, rather
than introducing those specialized types directly into the format.



Am Mo., 2. Nov. 2026 um 20:23 Uhr schrieb Jan Finis <jp...@gmail.com>:

> > Yes, I expect different functions in engines. I think pgvector uses more
> > specific functions for vectors, like `vector_dims` and `vector_norms`. As
> > far as I'm aware, these aren't available for arrays without an explicit
> > cast.
>
> pg_vector is IMHO just one example that cannot be generalized. The reason
> (I suspect) that it uses its own type is that PostgreSQL's array type is
> somewhat
> overly bloated. It is not fixed length and it even supports multiple
> dimensions, leading
> to an overly verbose binary format. Had PostgreSQL a fixed-length array
> type that
> was just represented as the values in binary, I think that pgvector would
> not have
> chosen an own type.
>
> In contrast, I would expect most engines that already have a fixed-length
> array type
> to just use that for vectors.
>
> Am Mo., 2. Nov. 2026 um 15:03 Uhr schrieb Rok Mihevc <rok.mihevc@gmail.com
> >:
>
>> > Could the proposal include a compatibility section addressing
>> unannotated
>> list data before we vote?
>>
>> Will, would a paragraph like this [1] be what you have in mind?
>>
>> On the topic of the vote - I would like to call it later today based on
>> parquet-format [2].
>>
>> [1]
>>
>> https://github.com/apache/parquet-format/pull/624/changes/8ad956d543e3074e1d9c8d4faa194199e06a8be4
>> [2] https://github.com/apache/parquet-format/pull/624
>>
>> Best,
>> Rok
>>
>> On Mon, Nov 2, 2026 at 12:42 PM Will Edwards via dev <
>> dev@parquet.apache.org>
>> wrote:
>>
>> > Thanks Ryan, Dan.
>> >
>> > My questions about what the type enables, and how you think Iceberg
>> tooling
>> > would use it, stem from the backward-compatibility concerns I've raised
>> > earlier in the thread. I was trying to weigh the cost against the
>> benefit.
>> > I worry that this annotation will fragment and orphan a lot of data:
>> > perfectly good vectors that become second-class just because they
>> weren't
>> > written with the annotation.
>> >
>> > There is an enormous amount of embedding data in Parquet today. Many
>> many
>> > lottabytes of it :) Some of it is packed in bespoke formats, but a very
>> > large share is plain, unannotated lists of floats or doubles. In
>> addition
>> > to all this existing data, many pipelines will keep writing new data
>> this
>> > way too, because they are unaware of the new annotation or can't be
>> > changed.
>> >
>> > Ryan, you said compatibility isn't a concern because this is a new
>> type. I
>> > think that assumes the data is new data written by an Iceberg writer.
>> But
>> > many Iceberg tables are built from files written by other engines. The
>> > standard implementations support this directly (add_files, migrate, name
>> > mapping), and the popular public embedding datasets are a typical
>> source.
>> > If Iceberg requires vector columns to carry the annotation, none of
>> those
>> > files can back a vector column without being rewritten, even though the
>> > bytes would be identical.
>> >
>> > It makes sense for Iceberg writers to check for inf/nan/null when they
>> > write a vector column. But since many files come from other writers,
>> robust
>> > readers should check correctness anyway, as they do with UTF-8 for
>> strings.
>> >
>> > So my position remains that this would be better done entirely in
>> Iceberg
>> > rather than pushed down to Parquet. Iceberg ought to accept unannotated
>> > LIST<FLOAT>/LIST<DOUBLE> files for vector columns, including via
>> add_files
>> > and a metadata-only list → vector promotion.
>> >
>> > Could the proposal include a compatibility section addressing
>> unannotated
>> > list data before we vote?
>> >
>> > Will
>> >
>> > On Sat, 31 Oct 2026 at 00:00, Ryan Blue via dev <dev@parquet.apache.org
>> >
>> > wrote:
>> >
>> > > > 1. do you expect that the backing files that are parquet MUST use
>> the
>> > new
>> > > logical vector type?
>> > >
>> > > I expect that the Iceberg spec will require writers to produce the
>> > > annotation.
>> > >
>> > > > 2. do you expect the software accessing  an iceberg table . . . to
>> > offer
>> > > different functions on vector columns compared to naked lists?
>> > >
>> > > Yes, I expect different functions in engines. I think pgvector uses
>> more
>> > > specific functions for vectors, like `vector_dims` and
>> `vector_norms`. As
>> > > far as I'm aware, these aren't available for arrays without an
>> explicit
>> > > cast.
>> > >
>> > > > Will a nearest neighbour search on a naked list of doubles be
>> > supported,
>> > > for example?
>> > >
>> > > This is an engine decision, but a list like this would not be exposed
>> as
>> > a
>> > > vector. You would most likely cast it to a vector in order to call
>> vector
>> > > functions.
>> > >
>> > > > 3. And will non-parquet backing files like avro and orc be supported
>> > for
>> > > the
>> > > vector columns?
>> > >
>> > > All Iceberg types can be stored in supported file formats.
>> > >
>> > > It seems like the direction for this line of questioning is basically
>> the
>> > > same as before: Can we just treat list<float> as a synonym of vector?
>> > >
>> > > And again, I think this is the wrong question. We have users right now
>> > that
>> > > struggle with errors that are found at read time, like the dimensions
>> of
>> > > their vectors not matching. Finding this out at read time is a problem
>> > and
>> > > we want to shift validation to write time so we can catch errors
>> earlier
>> > > and have guarantees that the data is usable. As I mentioned, one of
>> the
>> > > ways we have customers using data is sharing it with systems like
>> > pgvector
>> > > that also have these requirements. Just because you can think of a
>> vector
>> > > as a very special list of floats doesn't mean that write time
>> guarantees
>> > > are unreasonable or unnecessary.
>> > >
>> > > Ryan
>> > >
>> > >
>> > > On Thu, Oct 29, 2026 at 12:57 PM Will Edwards via dev <
>> > > dev@parquet.apache.org> wrote:
>> > >
>> > > > Thx for replying.
>> > > >
>> > > > I keep trying to explain that there are no performance advantages.
>> But
>> > > > anyway :)
>> > > >
>> > > > I understand you are envisaging a vector column type (with the fixed
>> > > length
>> > > > nan inf constraints you’ve been detailing) being added to iceberg?
>> > Then
>> > > > the following choices are interesting:
>> > > >
>> > > > 1. do you expect that the backing files that are parquet MUST use
>> the
>> > new
>> > > > logical vector type?
>> > > >
>> > > > 2. do you expect the software accessing  an iceberg table -
>> sparksql,
>> > for
>> > > > example - to offer different functions on vector columns compared to
>> > > naked
>> > > > lists?  Will a nearest neighbour search on a naked list of doubles
>> be
>> > > > supported, for example?
>> > > >
>> > > > 3. And will non-parquet backing files like avro and orc be supported
>> > for
>> > > > the vector columns?
>> > > >
>> > > > thx
>> > > >
>> > > > Will
>> > > >
>> > > > On Sat, 24 Oct 2026 at 02:08, Ryan Blue <bl...@apache.org> wrote:
>> > > >
>> > > > > > I'm asking for a workload that can't be done on LIST<FLOAT> now
>> and
>> > > > would
>> > > > work with VECTOR. Jan asked the same question, and I don't think
>> either
>> > > of
>> > > > us has seen one yet.
>> > > >
>> > > > I think this is the wrong question to ask, but I'm glad that you're
>> > > asking
>> > > > directly. Clearly, a variable-length list is a superset of a
>> > fixed-length
>> > > > list. And a list of floats is a superset of a list of finite
>> floats. A
>> > > list
>> > > > of optional elements is a superset of a list of required elements. I
>> > > don't
>> > > > think there's a use case that _can't_ be done.
>> > > >
>> > > > Is there a use case that can't be done with binary in place of
>> string?
>> > > And
>> > > > why don't we use JSON or Variant for everything? Just because we can
>> > use
>> > > a
>> > > > wider type doesn't mean that we don't benefit from communicating
>> > > guarantees
>> > > > like "this must be UTF-8".
>> > > >
>> > > > Types introduce guarantees that we can use to get better performance
>> > and
>> > > to
>> > > > either prevent errors or raise them closer to where data is
>> produced.
>> > > >
>> > > > For vectors, the guarantee that elements are finite gives us better
>> > > > performance because we don't need to check each element. This also
>> > moves
>> > > > the error closer to where data is produced because writers are
>> required
>> > > to
>> > > > verify the guarantees about each element and will detect bad values
>> at
>> > > > write time rather than at read time.
>> > > >
>> > > > Ryan
>> > > >
>> > >
>> >
>>
>

Re: [DISCUSS] Numeric Vector type: physical representation (outcome of focused call)

[Jan Finis <jp...@gmail.com>]

> Yes, I expect different functions in engines. I think pgvector uses more
> specific functions for vectors, like `vector_dims` and `vector_norms`. As
> far as I'm aware, these aren't available for arrays without an explicit
> cast.

pg_vector is IMHO just one example that cannot be generalized. The reason
(I suspect) that it uses its own type is that PostgreSQL's array type is
somewhat
overly bloated. It is not fixed length and it even supports multiple
dimensions, leading
to an overly verbose binary format. Had PostgreSQL a fixed-length array
type that
was just represented as the values in binary, I think that pgvector would
not have
chosen an own type.

In contrast, I would expect most engines that already have a fixed-length
array type
to just use that for vectors.

Am Mo., 2. Nov. 2026 um 15:03 Uhr schrieb Rok Mihevc <ro...@gmail.com>:

> > Could the proposal include a compatibility section addressing unannotated
> list data before we vote?
>
> Will, would a paragraph like this [1] be what you have in mind?
>
> On the topic of the vote - I would like to call it later today based on
> parquet-format [2].
>
> [1]
>
> https://github.com/apache/parquet-format/pull/624/changes/8ad956d543e3074e1d9c8d4faa194199e06a8be4
> [2] https://github.com/apache/parquet-format/pull/624
>
> Best,
> Rok
>
> On Mon, Nov 2, 2026 at 12:42 PM Will Edwards via dev <
> dev@parquet.apache.org>
> wrote:
>
> > Thanks Ryan, Dan.
> >
> > My questions about what the type enables, and how you think Iceberg
> tooling
> > would use it, stem from the backward-compatibility concerns I've raised
> > earlier in the thread. I was trying to weigh the cost against the
> benefit.
> > I worry that this annotation will fragment and orphan a lot of data:
> > perfectly good vectors that become second-class just because they weren't
> > written with the annotation.
> >
> > There is an enormous amount of embedding data in Parquet today. Many many
> > lottabytes of it :) Some of it is packed in bespoke formats, but a very
> > large share is plain, unannotated lists of floats or doubles. In addition
> > to all this existing data, many pipelines will keep writing new data this
> > way too, because they are unaware of the new annotation or can't be
> > changed.
> >
> > Ryan, you said compatibility isn't a concern because this is a new type.
> I
> > think that assumes the data is new data written by an Iceberg writer. But
> > many Iceberg tables are built from files written by other engines. The
> > standard implementations support this directly (add_files, migrate, name
> > mapping), and the popular public embedding datasets are a typical source.
> > If Iceberg requires vector columns to carry the annotation, none of those
> > files can back a vector column without being rewritten, even though the
> > bytes would be identical.
> >
> > It makes sense for Iceberg writers to check for inf/nan/null when they
> > write a vector column. But since many files come from other writers,
> robust
> > readers should check correctness anyway, as they do with UTF-8 for
> strings.
> >
> > So my position remains that this would be better done entirely in Iceberg
> > rather than pushed down to Parquet. Iceberg ought to accept unannotated
> > LIST<FLOAT>/LIST<DOUBLE> files for vector columns, including via
> add_files
> > and a metadata-only list → vector promotion.
> >
> > Could the proposal include a compatibility section addressing unannotated
> > list data before we vote?
> >
> > Will
> >
> > On Sat, 31 Oct 2026 at 00:00, Ryan Blue via dev <de...@parquet.apache.org>
> > wrote:
> >
> > > > 1. do you expect that the backing files that are parquet MUST use the
> > new
> > > logical vector type?
> > >
> > > I expect that the Iceberg spec will require writers to produce the
> > > annotation.
> > >
> > > > 2. do you expect the software accessing  an iceberg table . . . to
> > offer
> > > different functions on vector columns compared to naked lists?
> > >
> > > Yes, I expect different functions in engines. I think pgvector uses
> more
> > > specific functions for vectors, like `vector_dims` and `vector_norms`.
> As
> > > far as I'm aware, these aren't available for arrays without an explicit
> > > cast.
> > >
> > > > Will a nearest neighbour search on a naked list of doubles be
> > supported,
> > > for example?
> > >
> > > This is an engine decision, but a list like this would not be exposed
> as
> > a
> > > vector. You would most likely cast it to a vector in order to call
> vector
> > > functions.
> > >
> > > > 3. And will non-parquet backing files like avro and orc be supported
> > for
> > > the
> > > vector columns?
> > >
> > > All Iceberg types can be stored in supported file formats.
> > >
> > > It seems like the direction for this line of questioning is basically
> the
> > > same as before: Can we just treat list<float> as a synonym of vector?
> > >
> > > And again, I think this is the wrong question. We have users right now
> > that
> > > struggle with errors that are found at read time, like the dimensions
> of
> > > their vectors not matching. Finding this out at read time is a problem
> > and
> > > we want to shift validation to write time so we can catch errors
> earlier
> > > and have guarantees that the data is usable. As I mentioned, one of the
> > > ways we have customers using data is sharing it with systems like
> > pgvector
> > > that also have these requirements. Just because you can think of a
> vector
> > > as a very special list of floats doesn't mean that write time
> guarantees
> > > are unreasonable or unnecessary.
> > >
> > > Ryan
> > >
> > >
> > > On Thu, Oct 29, 2026 at 12:57 PM Will Edwards via dev <
> > > dev@parquet.apache.org> wrote:
> > >
> > > > Thx for replying.
> > > >
> > > > I keep trying to explain that there are no performance advantages.
> But
> > > > anyway :)
> > > >
> > > > I understand you are envisaging a vector column type (with the fixed
> > > length
> > > > nan inf constraints you’ve been detailing) being added to iceberg?
> > Then
> > > > the following choices are interesting:
> > > >
> > > > 1. do you expect that the backing files that are parquet MUST use the
> > new
> > > > logical vector type?
> > > >
> > > > 2. do you expect the software accessing  an iceberg table - sparksql,
> > for
> > > > example - to offer different functions on vector columns compared to
> > > naked
> > > > lists?  Will a nearest neighbour search on a naked list of doubles be
> > > > supported, for example?
> > > >
> > > > 3. And will non-parquet backing files like avro and orc be supported
> > for
> > > > the vector columns?
> > > >
> > > > thx
> > > >
> > > > Will
> > > >
> > > > On Sat, 24 Oct 2026 at 02:08, Ryan Blue <bl...@apache.org> wrote:
> > > >
> > > > > > I'm asking for a workload that can't be done on LIST<FLOAT> now
> and
> > > > would
> > > > work with VECTOR. Jan asked the same question, and I don't think
> either
> > > of
> > > > us has seen one yet.
> > > >
> > > > I think this is the wrong question to ask, but I'm glad that you're
> > > asking
> > > > directly. Clearly, a variable-length list is a superset of a
> > fixed-length
> > > > list. And a list of floats is a superset of a list of finite floats.
> A
> > > list
> > > > of optional elements is a superset of a list of required elements. I
> > > don't
> > > > think there's a use case that _can't_ be done.
> > > >
> > > > Is there a use case that can't be done with binary in place of
> string?
> > > And
> > > > why don't we use JSON or Variant for everything? Just because we can
> > use
> > > a
> > > > wider type doesn't mean that we don't benefit from communicating
> > > guarantees
> > > > like "this must be UTF-8".
> > > >
> > > > Types introduce guarantees that we can use to get better performance
> > and
> > > to
> > > > either prevent errors or raise them closer to where data is produced.
> > > >
> > > > For vectors, the guarantee that elements are finite gives us better
> > > > performance because we don't need to check each element. This also
> > moves
> > > > the error closer to where data is produced because writers are
> required
> > > to
> > > > verify the guarantees about each element and will detect bad values
> at
> > > > write time rather than at read time.
> > > >
> > > > Ryan
> > > >
> > >
> >
>

Re: [DISCUSS] Numeric Vector type: physical representation (outcome of focused call)

[Rok Mihevc <ro...@gmail.com>]

> Could the proposal include a compatibility section addressing unannotated
list data before we vote?

Will, would a paragraph like this [1] be what you have in mind?

On the topic of the vote - I would like to call it later today based on
parquet-format [2].

[1]
https://github.com/apache/parquet-format/pull/624/changes/8ad956d543e3074e1d9c8d4faa194199e06a8be4
[2] https://github.com/apache/parquet-format/pull/624

Best,
Rok

On Mon, Nov 2, 2026 at 12:42 PM Will Edwards via dev <de...@parquet.apache.org>
wrote:

> Thanks Ryan, Dan.
>
> My questions about what the type enables, and how you think Iceberg tooling
> would use it, stem from the backward-compatibility concerns I've raised
> earlier in the thread. I was trying to weigh the cost against the benefit.
> I worry that this annotation will fragment and orphan a lot of data:
> perfectly good vectors that become second-class just because they weren't
> written with the annotation.
>
> There is an enormous amount of embedding data in Parquet today. Many many
> lottabytes of it :) Some of it is packed in bespoke formats, but a very
> large share is plain, unannotated lists of floats or doubles. In addition
> to all this existing data, many pipelines will keep writing new data this
> way too, because they are unaware of the new annotation or can't be
> changed.
>
> Ryan, you said compatibility isn't a concern because this is a new type. I
> think that assumes the data is new data written by an Iceberg writer. But
> many Iceberg tables are built from files written by other engines. The
> standard implementations support this directly (add_files, migrate, name
> mapping), and the popular public embedding datasets are a typical source.
> If Iceberg requires vector columns to carry the annotation, none of those
> files can back a vector column without being rewritten, even though the
> bytes would be identical.
>
> It makes sense for Iceberg writers to check for inf/nan/null when they
> write a vector column. But since many files come from other writers, robust
> readers should check correctness anyway, as they do with UTF-8 for strings.
>
> So my position remains that this would be better done entirely in Iceberg
> rather than pushed down to Parquet. Iceberg ought to accept unannotated
> LIST<FLOAT>/LIST<DOUBLE> files for vector columns, including via add_files
> and a metadata-only list → vector promotion.
>
> Could the proposal include a compatibility section addressing unannotated
> list data before we vote?
>
> Will
>
> On Sat, 31 Oct 2026 at 00:00, Ryan Blue via dev <de...@parquet.apache.org>
> wrote:
>
> > > 1. do you expect that the backing files that are parquet MUST use the
> new
> > logical vector type?
> >
> > I expect that the Iceberg spec will require writers to produce the
> > annotation.
> >
> > > 2. do you expect the software accessing  an iceberg table . . . to
> offer
> > different functions on vector columns compared to naked lists?
> >
> > Yes, I expect different functions in engines. I think pgvector uses more
> > specific functions for vectors, like `vector_dims` and `vector_norms`. As
> > far as I'm aware, these aren't available for arrays without an explicit
> > cast.
> >
> > > Will a nearest neighbour search on a naked list of doubles be
> supported,
> > for example?
> >
> > This is an engine decision, but a list like this would not be exposed as
> a
> > vector. You would most likely cast it to a vector in order to call vector
> > functions.
> >
> > > 3. And will non-parquet backing files like avro and orc be supported
> for
> > the
> > vector columns?
> >
> > All Iceberg types can be stored in supported file formats.
> >
> > It seems like the direction for this line of questioning is basically the
> > same as before: Can we just treat list<float> as a synonym of vector?
> >
> > And again, I think this is the wrong question. We have users right now
> that
> > struggle with errors that are found at read time, like the dimensions of
> > their vectors not matching. Finding this out at read time is a problem
> and
> > we want to shift validation to write time so we can catch errors earlier
> > and have guarantees that the data is usable. As I mentioned, one of the
> > ways we have customers using data is sharing it with systems like
> pgvector
> > that also have these requirements. Just because you can think of a vector
> > as a very special list of floats doesn't mean that write time guarantees
> > are unreasonable or unnecessary.
> >
> > Ryan
> >
> >
> > On Thu, Oct 29, 2026 at 12:57 PM Will Edwards via dev <
> > dev@parquet.apache.org> wrote:
> >
> > > Thx for replying.
> > >
> > > I keep trying to explain that there are no performance advantages. But
> > > anyway :)
> > >
> > > I understand you are envisaging a vector column type (with the fixed
> > length
> > > nan inf constraints you’ve been detailing) being added to iceberg?
> Then
> > > the following choices are interesting:
> > >
> > > 1. do you expect that the backing files that are parquet MUST use the
> new
> > > logical vector type?
> > >
> > > 2. do you expect the software accessing  an iceberg table - sparksql,
> for
> > > example - to offer different functions on vector columns compared to
> > naked
> > > lists?  Will a nearest neighbour search on a naked list of doubles be
> > > supported, for example?
> > >
> > > 3. And will non-parquet backing files like avro and orc be supported
> for
> > > the vector columns?
> > >
> > > thx
> > >
> > > Will
> > >
> > > On Sat, 24 Oct 2026 at 02:08, Ryan Blue <bl...@apache.org> wrote:
> > >
> > > > > I'm asking for a workload that can't be done on LIST<FLOAT> now and
> > > would
> > > work with VECTOR. Jan asked the same question, and I don't think either
> > of
> > > us has seen one yet.
> > >
> > > I think this is the wrong question to ask, but I'm glad that you're
> > asking
> > > directly. Clearly, a variable-length list is a superset of a
> fixed-length
> > > list. And a list of floats is a superset of a list of finite floats. A
> > list
> > > of optional elements is a superset of a list of required elements. I
> > don't
> > > think there's a use case that _can't_ be done.
> > >
> > > Is there a use case that can't be done with binary in place of string?
> > And
> > > why don't we use JSON or Variant for everything? Just because we can
> use
> > a
> > > wider type doesn't mean that we don't benefit from communicating
> > guarantees
> > > like "this must be UTF-8".
> > >
> > > Types introduce guarantees that we can use to get better performance
> and
> > to
> > > either prevent errors or raise them closer to where data is produced.
> > >
> > > For vectors, the guarantee that elements are finite gives us better
> > > performance because we don't need to check each element. This also
> moves
> > > the error closer to where data is produced because writers are required
> > to
> > > verify the guarantees about each element and will detect bad values at
> > > write time rather than at read time.
> > >
> > > Ryan
> > >
> >
>

Re: [DISCUSS] Numeric Vector type: physical representation (outcome of focused call)

[Will Edwards via dev <de...@parquet.apache.org>]

Thanks Ryan, Dan.

My questions about what the type enables, and how you think Iceberg tooling
would use it, stem from the backward-compatibility concerns I've raised
earlier in the thread. I was trying to weigh the cost against the benefit.
I worry that this annotation will fragment and orphan a lot of data:
perfectly good vectors that become second-class just because they weren't
written with the annotation.

There is an enormous amount of embedding data in Parquet today. Many many
lottabytes of it :) Some of it is packed in bespoke formats, but a very
large share is plain, unannotated lists of floats or doubles. In addition
to all this existing data, many pipelines will keep writing new data this
way too, because they are unaware of the new annotation or can't be changed.

Ryan, you said compatibility isn't a concern because this is a new type. I
think that assumes the data is new data written by an Iceberg writer. But
many Iceberg tables are built from files written by other engines. The
standard implementations support this directly (add_files, migrate, name
mapping), and the popular public embedding datasets are a typical source.
If Iceberg requires vector columns to carry the annotation, none of those
files can back a vector column without being rewritten, even though the
bytes would be identical.

It makes sense for Iceberg writers to check for inf/nan/null when they
write a vector column. But since many files come from other writers, robust
readers should check correctness anyway, as they do with UTF-8 for strings.

So my position remains that this would be better done entirely in Iceberg
rather than pushed down to Parquet. Iceberg ought to accept unannotated
LIST<FLOAT>/LIST<DOUBLE> files for vector columns, including via add_files
and a metadata-only list → vector promotion.

Could the proposal include a compatibility section addressing unannotated
list data before we vote?

Will

On Sat, 31 Oct 2026 at 00:00, Ryan Blue via dev <de...@parquet.apache.org>
wrote:

> > 1. do you expect that the backing files that are parquet MUST use the new
> logical vector type?
>
> I expect that the Iceberg spec will require writers to produce the
> annotation.
>
> > 2. do you expect the software accessing  an iceberg table . . . to offer
> different functions on vector columns compared to naked lists?
>
> Yes, I expect different functions in engines. I think pgvector uses more
> specific functions for vectors, like `vector_dims` and `vector_norms`. As
> far as I'm aware, these aren't available for arrays without an explicit
> cast.
>
> > Will a nearest neighbour search on a naked list of doubles be supported,
> for example?
>
> This is an engine decision, but a list like this would not be exposed as a
> vector. You would most likely cast it to a vector in order to call vector
> functions.
>
> > 3. And will non-parquet backing files like avro and orc be supported for
> the
> vector columns?
>
> All Iceberg types can be stored in supported file formats.
>
> It seems like the direction for this line of questioning is basically the
> same as before: Can we just treat list<float> as a synonym of vector?
>
> And again, I think this is the wrong question. We have users right now that
> struggle with errors that are found at read time, like the dimensions of
> their vectors not matching. Finding this out at read time is a problem and
> we want to shift validation to write time so we can catch errors earlier
> and have guarantees that the data is usable. As I mentioned, one of the
> ways we have customers using data is sharing it with systems like pgvector
> that also have these requirements. Just because you can think of a vector
> as a very special list of floats doesn't mean that write time guarantees
> are unreasonable or unnecessary.
>
> Ryan
>
>
> On Thu, Oct 29, 2026 at 12:57 PM Will Edwards via dev <
> dev@parquet.apache.org> wrote:
>
> > Thx for replying.
> >
> > I keep trying to explain that there are no performance advantages. But
> > anyway :)
> >
> > I understand you are envisaging a vector column type (with the fixed
> length
> > nan inf constraints you’ve been detailing) being added to iceberg?  Then
> > the following choices are interesting:
> >
> > 1. do you expect that the backing files that are parquet MUST use the new
> > logical vector type?
> >
> > 2. do you expect the software accessing  an iceberg table - sparksql, for
> > example - to offer different functions on vector columns compared to
> naked
> > lists?  Will a nearest neighbour search on a naked list of doubles be
> > supported, for example?
> >
> > 3. And will non-parquet backing files like avro and orc be supported for
> > the vector columns?
> >
> > thx
> >
> > Will
> >
> > On Sat, 24 Oct 2026 at 02:08, Ryan Blue <bl...@apache.org> wrote:
> >
> > > > I'm asking for a workload that can't be done on LIST<FLOAT> now and
> > would
> > work with VECTOR. Jan asked the same question, and I don't think either
> of
> > us has seen one yet.
> >
> > I think this is the wrong question to ask, but I'm glad that you're
> asking
> > directly. Clearly, a variable-length list is a superset of a fixed-length
> > list. And a list of floats is a superset of a list of finite floats. A
> list
> > of optional elements is a superset of a list of required elements. I
> don't
> > think there's a use case that _can't_ be done.
> >
> > Is there a use case that can't be done with binary in place of string?
> And
> > why don't we use JSON or Variant for everything? Just because we can use
> a
> > wider type doesn't mean that we don't benefit from communicating
> guarantees
> > like "this must be UTF-8".
> >
> > Types introduce guarantees that we can use to get better performance and
> to
> > either prevent errors or raise them closer to where data is produced.
> >
> > For vectors, the guarantee that elements are finite gives us better
> > performance because we don't need to check each element. This also moves
> > the error closer to where data is produced because writers are required
> to
> > verify the guarantees about each element and will detect bad values at
> > write time rather than at read time.
> >
> > Ryan
> >
>

Re: [DISCUSS] Numeric Vector type: physical representation (outcome of focused call)

[Ryan Blue via dev <de...@parquet.apache.org>]

> 1. do you expect that the backing files that are parquet MUST use the new
logical vector type?

I expect that the Iceberg spec will require writers to produce the
annotation.

> 2. do you expect the software accessing  an iceberg table . . . to offer
different functions on vector columns compared to naked lists?

Yes, I expect different functions in engines. I think pgvector uses more
specific functions for vectors, like `vector_dims` and `vector_norms`. As
far as I'm aware, these aren't available for arrays without an explicit
cast.

> Will a nearest neighbour search on a naked list of doubles be supported,
for example?

This is an engine decision, but a list like this would not be exposed as a
vector. You would most likely cast it to a vector in order to call vector
functions.

> 3. And will non-parquet backing files like avro and orc be supported for the
vector columns?

All Iceberg types can be stored in supported file formats.

It seems like the direction for this line of questioning is basically the
same as before: Can we just treat list<float> as a synonym of vector?

And again, I think this is the wrong question. We have users right now that
struggle with errors that are found at read time, like the dimensions of
their vectors not matching. Finding this out at read time is a problem and
we want to shift validation to write time so we can catch errors earlier
and have guarantees that the data is usable. As I mentioned, one of the
ways we have customers using data is sharing it with systems like pgvector
that also have these requirements. Just because you can think of a vector
as a very special list of floats doesn't mean that write time guarantees
are unreasonable or unnecessary.

Ryan


On Thu, Oct 29, 2026 at 12:57 PM Will Edwards via dev <
dev@parquet.apache.org> wrote:

> Thx for replying.
>
> I keep trying to explain that there are no performance advantages. But
> anyway :)
>
> I understand you are envisaging a vector column type (with the fixed length
> nan inf constraints you’ve been detailing) being added to iceberg?  Then
> the following choices are interesting:
>
> 1. do you expect that the backing files that are parquet MUST use the new
> logical vector type?
>
> 2. do you expect the software accessing  an iceberg table - sparksql, for
> example - to offer different functions on vector columns compared to naked
> lists?  Will a nearest neighbour search on a naked list of doubles be
> supported, for example?
>
> 3. And will non-parquet backing files like avro and orc be supported for
> the vector columns?
>
> thx
>
> Will
>
> On Sat, 24 Oct 2026 at 02:08, Ryan Blue <bl...@apache.org> wrote:
>
> > > I'm asking for a workload that can't be done on LIST<FLOAT> now and
> would
> work with VECTOR. Jan asked the same question, and I don't think either of
> us has seen one yet.
>
> I think this is the wrong question to ask, but I'm glad that you're asking
> directly. Clearly, a variable-length list is a superset of a fixed-length
> list. And a list of floats is a superset of a list of finite floats. A list
> of optional elements is a superset of a list of required elements. I don't
> think there's a use case that _can't_ be done.
>
> Is there a use case that can't be done with binary in place of string? And
> why don't we use JSON or Variant for everything? Just because we can use a
> wider type doesn't mean that we don't benefit from communicating guarantees
> like "this must be UTF-8".
>
> Types introduce guarantees that we can use to get better performance and to
> either prevent errors or raise them closer to where data is produced.
>
> For vectors, the guarantee that elements are finite gives us better
> performance because we don't need to check each element. This also moves
> the error closer to where data is produced because writers are required to
> verify the guarantees about each element and will detect bad values at
> write time rather than at read time.
>
> Ryan
>

Re: [DISCUSS] Numeric Vector type: physical representation (outcome of focused call)

[Daniel Weeks <dw...@apache.org>]

Let's stop worrying about performance aspects of this.  If people find
optimizations, that's great, but it's not the intent behind the addition of
this type.

Yes, both Iceberg and Delta table formats are interested in adding this
type with the specific semantics we've discussed.

In Iceberg we generally define the Iceberg type to file format type mapping
<https://iceberg.apache.org/spec/#parquet> and strongly prefer to have a
representative logical type.  Avro and ORC are supported to a lesser extent
due to how they're used and the level of support available across
implementations.  I wouldn't worry about the other formats as we prefer to
lean into parquet for consistency and ecosystem compatibility.

As to how it's used, that's not really for Parquet or Iceberg to define.
It's entirely possible there will be built-in functions or UDFs that allow
or return only Vector defined types.  Others could support both, but
functions are largely left to engine implementations to define, as are
runtime management and casting behaviors.

At this point, I think we've done a good job of narrowing this type
definition to 'FINITE_VECTOR' to both clarify the semantics and deconflict
it with other potential future solutions that have different behaiors or
use cases.  This representation is also aligned with other vector
implementations and with proposals from upstream projects that want to
leverage this type.

Will, I understand you may have different use cases, but we shouldn't be
blocking progress here.  There's substantial agreement that this is a
useful addition and if you're looking for something different, please make
a proposal that addresses your needs.  This proposal doesn't need to
address every use case; it's well defined and narrowly scoped.  I think
it's more important to make meaningful progress than try to solve for
every related use case.

Other proposals are more than welcome and expected as we get more examples
of well defined types (embeddings, tensors, etc).

I think the next steps are for Rok to update the proposal and then I'd urge
the community to move forward with a vote on the more narrowly defined
type.

-Dan

On Thu, Oct 29, 2026 at 12:57 PM Will Edwards via dev <
dev@parquet.apache.org> wrote:

> Thx for replying.
>
> I keep trying to explain that there are no performance advantages. But
> anyway :)
>
> I understand you are envisaging a vector column type (with the fixed length
> nan inf constraints you’ve been detailing) being added to iceberg?  Then
> the following choices are interesting:
>
> 1. do you expect that the backing files that are parquet MUST use the new
> logical vector type?
>
> 2. do you expect the software accessing  an iceberg table - sparksql, for
> example - to offer different functions on vector columns compared to naked
> lists?  Will a nearest neighbour search on a naked list of doubles be
> supported, for example?
>
> 3. And will non-parquet backing files like avro and orc be supported for
> the vector columns?
>
> thx
>
> Will
>
> On Sat, 24 Oct 2026 at 02:08, Ryan Blue <bl...@apache.org> wrote:
>
> > > I'm asking for a workload that can't be done on LIST<FLOAT> now and
> would
> work with VECTOR. Jan asked the same question, and I don't think either of
> us has seen one yet.
>
> I think this is the wrong question to ask, but I'm glad that you're asking
> directly. Clearly, a variable-length list is a superset of a fixed-length
> list. And a list of floats is a superset of a list of finite floats. A list
> of optional elements is a superset of a list of required elements. I don't
> think there's a use case that _can't_ be done.
>
> Is there a use case that can't be done with binary in place of string? And
> why don't we use JSON or Variant for everything? Just because we can use a
> wider type doesn't mean that we don't benefit from communicating guarantees
> like "this must be UTF-8".
>
> Types introduce guarantees that we can use to get better performance and to
> either prevent errors or raise them closer to where data is produced.
>
> For vectors, the guarantee that elements are finite gives us better
> performance because we don't need to check each element. This also moves
> the error closer to where data is produced because writers are required to
> verify the guarantees about each element and will detect bad values at
> write time rather than at read time.
>
> Ryan
>

Re: [DISCUSS] Numeric Vector type: physical representation (outcome of focused call)

[Will Edwards via dev <de...@parquet.apache.org>]

Thx for replying.

I keep trying to explain that there are no performance advantages. But
anyway :)

I understand you are envisaging a vector column type (with the fixed length
nan inf constraints you’ve been detailing) being added to iceberg?  Then
the following choices are interesting:

1. do you expect that the backing files that are parquet MUST use the new
logical vector type?

2. do you expect the software accessing  an iceberg table - sparksql, for
example - to offer different functions on vector columns compared to naked
lists?  Will a nearest neighbour search on a naked list of doubles be
supported, for example?

3. And will non-parquet backing files like avro and orc be supported for
the vector columns?

thx

Will

On Sat, 24 Oct 2026 at 02:08, Ryan Blue <bl...@apache.org> wrote:

> > I'm asking for a workload that can't be done on LIST<FLOAT> now and would
work with VECTOR. Jan asked the same question, and I don't think either of
us has seen one yet.

I think this is the wrong question to ask, but I'm glad that you're asking
directly. Clearly, a variable-length list is a superset of a fixed-length
list. And a list of floats is a superset of a list of finite floats. A list
of optional elements is a superset of a list of required elements. I don't
think there's a use case that _can't_ be done.

Is there a use case that can't be done with binary in place of string? And
why don't we use JSON or Variant for everything? Just because we can use a
wider type doesn't mean that we don't benefit from communicating guarantees
like "this must be UTF-8".

Types introduce guarantees that we can use to get better performance and to
either prevent errors or raise them closer to where data is produced.

For vectors, the guarantee that elements are finite gives us better
performance because we don't need to check each element. This also moves
the error closer to where data is produced because writers are required to
verify the guarantees about each element and will detect bad values at
write time rather than at read time.

Ryan

Re: [DISCUSS] Numeric Vector type: physical representation (outcome of focused call)

[Ryan Blue <bl...@apache.org>]

> I'm asking for a workload that can't be done on LIST<FLOAT> now and would
work with VECTOR. Jan asked the same question, and I don't think either of
us has seen one yet.

I think this is the wrong question to ask, but I'm glad that you're asking
directly. Clearly, a variable-length list is a superset of a fixed-length
list. And a list of floats is a superset of a list of finite floats. A list
of optional elements is a superset of a list of required elements. I don't
think there's a use case that _can't_ be done.

Is there a use case that can't be done with binary in place of string? And
why don't we use JSON or Variant for everything? Just because we can use a
wider type doesn't mean that we don't benefit from communicating guarantees
like "this must be UTF-8".

Types introduce guarantees that we can use to get better performance and to
either prevent errors or raise them closer to where data is produced.

For vectors, the guarantee that elements are finite gives us better
performance because we don't need to check each element. This also moves
the error closer to where data is produced because writers are required to
verify the guarantees about each element and will detect bad values at
write time rather than at read time.

Ryan

On Thu, Oct 22, 2026 at 7:55 AM Fokko Driesprong <fo...@driesprong.frl>
wrote:

> It feels like we're running in circles, let me add my 2c.
>
> > I'd like to put the vector type on hold in Parquet until there's a
> general proposal for ML tensors and sparse, packed or quantised
> representations. That's where a new type would let engines do interoperable
> things they can't do today.
>
> I'm afraid that's not how it works. There are folks that are willing to do
> the legwork for the VECTOR logical type. Sparse and Quantised
> representation sounds more like something where we want to focus on
> encoding, rather than a logical type.
>
> > I'm asking for a workload that can't be done on LIST<FLOAT> now and
> would work with VECTOR.
>
> We're all smart engineers, and we can make it work. But I think this is
> more about enforcing certain constraints on the data. Similar to the
> example of UTF8 and BYTES that was coined earlier. Or you can store UUIDs
> in a FIXED_LEN_BYTE_ARRAY, but there is a specific annotation for that as
> well. It is nice to have some semantical meaning to the types.
>
> > If a finite fixed-length vector contract is useful to table formats,
> perhaps Iceberg is the place for it
>
> I strongly feel that Parquet should be self-contained. As with the
> examples above, I feel this is something that is in line with Parquet.
>
> Kind regards,
> Fokko
>
>
> Sent with Proton Mail secure email.
>
> On Thursday, October 22nd, 2026 at 07:11, Will Edwards via dev <
> dev@parquet.apache.org> wrote:
>
> > Hi Ryan,
> >
> > Thanks for the detailed replies. Let me restate my position more
> directly,
> > since I don't think I made it clear.
> >
> > I'd like to put the vector type on hold in Parquet until there's a
> general
> > proposal for ML tensors and sparse, packed or quantised representations.
> > That's where a new type would let engines do interoperable things they
> > can't do today.
> >
> > My earlier question was what VECTOR makes possible that isn't possible
> > today. The answers so far describe what the annotation asserts: that the
> > data is a vector, that it has a fixed length, and that its values are
> > finite and non-null. I'm asking for a workload that can't be done on
> > LIST<FLOAT> now and would work with VECTOR. Jan asked the same question,
> > and I don't think either of us has seen one yet.
> >
> > For embeddings, LIST<FLOAT> is what is already widely used today, and it
> > works. My compatibility concern is about writers. A tool that doesn't
> know
> > VECTOR rewrites the file as plain LIST<FLOAT>. If engines then require
> the
> > annotation, that data stops working. If they don't require it, the
> > annotation adds little. A world where VECTOR exists and plain LIST<FLOAT>
> > still works fully for ANN etc is a world that doesn't need VECTOR.
> >
> > On optimisations, Dan framed this as semantics rather than performance,
> but
> > you've listed decoding optimisations. In my experience readers can remove
> > the cost of variable-length lists without any format change, and I'm
> happy
> > to share the recipes with anyone interested. If fixed length is still the
> > real gain, a fixed-size list gives it to everyone, NaN or not.
> >
> > If table formats need a finite-vector contract, I think the table layer
> is
> > a better home than the file format. An Iceberg type would apply equally
> to
> > Parquet, Avro and ORC. I'd still be wary of Iceberg rejecting untagged
> > columns, but it's a better place for that debate than Parquet.
> >
> > Best,
> > Will
> >
> > On Thu, 22 Oct 2026 at 02:00, Ryan Blue <bl...@apache.org> wrote:
> >
> > > Thanks, Curt. I wasn't aware of that behavior in MySQL. I was trying to
> > > learn more and I don't see it in documentation either. Do you have any
> more
> > > information about how MySQL handles this? One page makes it sound like
> it
> > > may be platform dependent
> > > <https://dev.mysql.com/doc/refman/26.7/en/problems-with-float.html>,
> where
> > > some values produce "inf" and some produce "0".
> > >
> > > My original comment was referring to databases that allow users to
> specify
> > > types that exclude non-finite values, which is what we would need in
> order
> > > to support a finite float in table formats since we already support the
> > > full range of values. If you know of any cases like that rather than
> just
> > > limiting values, I'm curious. Thanks!
> > >
> > > Ryan
> > >
> > > On Wed, Oct 21, 2026 at 3:48 PM Curt Hagenlocher <curt@hagenlocher.org
> >
> > > wrote:
> > >
> > > > > I'm not aware of databases with finite floating point types.
> > > >
> > > > Microsoft SQL Server, MySQL and SAP ASE (née Sybase) do not allow
> > > > storing non-finite floating-point values in their binary
> > > > floating-point columns. They don't have a separate type which does
> > > > allow this.
> > > >
> > > > On Wed, Oct 21, 2026 at 3:30 PM Ryan Blue <bl...@apache.org> wrote:
> > > > >
> > > > > Jan,
> > > > >
> > > > > I agree that we _could_ model vectors as an orthogonal finiteness
> > > > property
> > > > > for elements in a fixed-length list. I don't know how useful that
> would
> > > > be.
> > > > > I've not had anyone ask for that, and I'm not aware of databases
> with
> > > > > finite floating point types. I think it makes more sense to move
> > > forward
> > > > > with the existing FINITE_VECTOR that is proposed.
> > > > >
> > > > > For the logical type argument, it sounds like the issue is that you
> > > want
> > > > to
> > > > > fall back to readers treating vectors as LIST. I don't see a good
> way
> > > to
> > > > > make that happen, but it also isn't needed by the table format use
> case
> > > > > because we are introducing this as a new type. Do you have a use
> case
> > > > where
> > > > > you are going to mix FINITE_VECTOR with LIST files and this will
> cause
> > > a
> > > > > problem?
> > > > >
> > > > > I don't think that we've had an expectation for logical types in
> the
> > > past
> > > > > that they produce a particular in-memory representation. Maybe we
> could
> > > > > allow setting both `LIST` and `FINITE_VECTOR` logical types at the
> same
> > > > > time? This kind of compatibility hasn't been a priority as far as
> I'm
> > > > > aware, but if it is important to you we can consider it.
> > > > >
> > > > > Ryan
> > > > >
> > > > > On Wed, Oct 21, 2026 at 1:13 PM Jan Finis <jp...@gmail.com>
> wrote:
> > > > >
> > > > > >  assuming what will be proposed is a new logical type
> FINITE_VECTOR
> > > > that
> > > > > > annotates the same column that would have been annotated as LIST
> > > > before:
> > > > > >
> > > > > > What I find somewhat suboptimal of FINITE_VECTOR is that
> finiteness
> > > is
> > > > part
> > > > > > of the vector type. Isn't the finiteness rather a property of the
> > > > contained
> > > > > > floats? Such a modeling would be more composable, as it would for
> > > > example
> > > > > > also allow to have normal (non-vector) float columns that are
> > > > restricted to
> > > > > > be finite. So, if we do think that we want finiteness in the type
> > > > system, I
> > > > > > would rather cut it into two parts: A fixed size vector concept,
> and
> > > a
> > > > new
> > > > > > way of labeling float columns finite. The vector type would then
> just
> > > > boil
> > > > > > down to be a fixed size list / we could just give the list type
> an
> > > > optional
> > > > > > fixed size field. Then, the column contained within it would be a
> > > float
> > > > > > tagged as finite.
> > > > > >
> > > > > > It was mentioned that a logical type annotating a list of float
> would
> > > > be
> > > > > > forward compatible, but - depending on the modeling - that is not
> > > > > > necessarily true: as logical types in Parquet do not support
> > > > subtyping, a
> > > > > > column cannot be a list of float and a (finite) vector at the
> same
> > > > time,
> > > > > > but only one of the two. It depends on how the type will be
> modeled
> > > in
> > > > > > thrift in the end, but if it is a new union member of
> `LogicalType`,
> > > > then
> > > > > > legacy engines will not understand that such a vector can also
> just
> > > be
> > > > > > interpreted as a list of floats. They will just see a nested
> column
> > > > type of
> > > > > > a logical type they don't understand, so they will likely reject
> > > > reading
> > > > > > this column (at least that's what the engine I'm maintaining
> would be
> > > > > > doing).
> > > > > >
> > > > > > In contrast, if we just give the list type an optional fixed
> size and
> > > > float
> > > > > > an optional finiteness, then the resulting thrift is actually
> fully
> > > > forward
> > > > > > compatible: Legacy engines will just ignore the fixed length and
> > > treat
> > > > the
> > > > > > list as variable length and they will ignore the finiteness and
> treat
> > > > the
> > > > > > element type as float.
> > > > > >
> > > > > > (A side note, which most people here may already be aware of, but
> > > it's
> > > > good
> > > > > > to repeat it: The not-NULL property doesn't need new type
> support at
> > > > all.
> > > > > > The normal type system of list already allows setting the
> contained
> > > > element
> > > > > > type to be optional or required. If it is set to be required,
> then
> > > > NULLs
> > > > > > are not allowed.)
> > > > > >
> > > > > > Cheers,
> > > > > > Jan
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > Am Di., 20. Okt. 2026 um 20:37 Uhr schrieb Russell Spitzer <
> > > > > > russell.spitzer@gmail.com>:
> > > > > >
> > > > > > > I support proceeding with the two types. I honestly think the
> cost
> > > of
> > > > > > > adding new logical types is overestimated. If we have as many
> > > > interested
> > > > > > > parties as have appeared in this thread we already know there
> is a
> > > > large
> > > > > > > group of engineers ready to do the work and engines ready to
> > > consume
> > > > (and
> > > > > > > produce) the type. The fact that implementations currently use
> > > > > > heuristics,
> > > > > > > sampling (I loved Gunnar's talk on his hardwood work on this
> at CoC
> > > > > > > Glasgow) or custom annotations to achieve the same performance
> > > > benefits
> > > > > > > makes me think we should just create the type and move on.
> > > > > > >
> > > > > > > On Mon, Oct 19, 2026 at 7:21 PM Ryan Blue <bl...@apache.org>
> wrote:
> > > > > > >
> > > > > > > > An annotation on list<float> is useful because:
> > > > > > > > - We want to correctly annotate what is being stored
> > > > > > > > - We want to unlock optimizations when each non-null list
> has the
> > > > same
> > > > > > > > number of elements
> > > > > > > > - We want to have writer guarantees
> > > > > > > >
> > > > > > > > I don't think that logical type compatibility is a good
> argument.
> > > > We
> > > > > > want
> > > > > > > > to avoid a case where we have to treat all binary like a
> string.
> > > > And
> > > > > > > > because we are introducing this in table formats after
> > > > standardizing it
> > > > > > > > here, we can do that reliably.
> > > > > > > >
> > > > > > > > As I said in my last email, I think there's a difference
> between
> > > > > > > finiteness
> > > > > > > > and general "invariants" you might want. Vectors aren't
> vectors
> > > if
> > > > they
> > > > > > > > don't have a length and a direction. Other invariants don't
> work
> > > > the
> > > > > > same
> > > > > > > > way. A zero-length vector is still a valid vector and can be
> > > used,
> > > > just
> > > > > > > not
> > > > > > > > for certain distance metrics.
> > > > > > > >
> > > > > > > > I also want to clarify that when we looked into this more
> closely
> > > > and
> > > > > > > > recognized the mismatch between the two separate use cases, I
> > > > offered
> > > > > > to
> > > > > > > > split out the finite vector work so that we didn't cause any
> > > > headaches
> > > > > > > for
> > > > > > > > people who are interested in fixed-length list. It sounds
> like
> > > not
> > > > > > > > splitting out the use cases is causing a lot of headache.
> Vectors
> > > > for
> > > > > > our
> > > > > > > > use case need a finiteness guarantee at write time, and
> > > > fixed-length
> > > > > > > lists
> > > > > > > > need to store NaN and other values.
> > > > > > > >
> > > > > > > > It's okay to have two types and we should for two different
> use
> > > > cases.
> > > > > > > > Otherwise, we are forcing one to make an unnecessary
> compromise.
> > > I
> > > > > > don't
> > > > > > > > want to force everyone to exclude NaN and infinite values
> from
> > > > > > > fixed-length
> > > > > > > > lists. Seems reasonable that we should also be able to have
> > > vectors
> > > > > > that
> > > > > > > > exclude those values, since that is how they are used and we
> have
> > > > to
> > > > > > also
> > > > > > > > guarantee it for exchanging data with other existing
> databases.
> > > > > > > >
> > > > > > > > Ryan
> > > > > > > >
> > > > > > > > On Mon, Oct 19, 2026 at 3:12 AM Will Edwards via dev <
> > > > > > > > dev@parquet.apache.org>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > It reads well. Three small fixes:
> > > > > > > > >
> > > > > > > > > 1. **"back in May or so"** is ambiguous. The
> > > > > > > fixed-size-list-for-tensors
> > > > > > > > > framing is from Rok's opening post in May 2024. The May
> 2026
> > > > design
> > > > > > doc
> > > > > > > > was
> > > > > > > > > already mostly about performance and layout. "Originally"
> > > avoids
> > > > > > anyone
> > > > > > > > > correcting the date.
> > > > > > > > > 2. **"Spark's sparse feature vectors routinely use NaN"**
> isn't
> > > > quite
> > > > > > > > > right. The NaN-for-missing behaviour comes from
> > > > `VectorAssembler` and
> > > > > > > > > `Imputer` and applies to Spark ML vectors generally, dense
> or
> > > > sparse.
> > > > > > > > > "Spark's feature vectors, dense and sparse," fixes it.
> > > > > > > > > 3. There are typos: "Iirc" should be "IIRC", and "Spark,
> > > > BigQuery and
> > > > > > > > > DuckDB and pretty much…" has a doubled "and".
> > > > > > > > >
> > > > > > > > > The last sentence of the invariants paragraph could be
> clearer.
> > > > > > > "Although
> > > > > > > > > that doesn't mean readers shouldn't check on read" has a
> double
> > > > > > > negative
> > > > > > > > > and is easy to misread. "Though careful readers will still
> > > check
> > > > on
> > > > > > > read"
> > > > > > > > > is simpler.
> > > > > > > > >
> > > > > > > > > With those applied:
> > > > > > > > >
> > > > > > > > > ---
> > > > > > > > >
> > > > > > > > > Hi all,
> > > > > > > > >
> > > > > > > > > The scope of this discussion keeps moving :). IIRC it
> > > originally
> > > > was
> > > > > > > > about
> > > > > > > > > fixed-size lists for ML tensors, became a performance
> question,
> > > > and
> > > > > > is
> > > > > > > > now
> > > > > > > > > about whether vectors are mathematically valid. Each step
> made
> > > > sense,
> > > > > > > > but I
> > > > > > > > > don't think we have consensus on where we've ended up?
> > > > > > > > >
> > > > > > > > > I think there is much useful work in what we left behind.
> Many
> > > ML
> > > > > > > vectors
> > > > > > > > > aren't embeddings for search. E.g. Spark's feature vectors,
> > > > dense and
> > > > > > > > > sparse, routinely use NaN to mark missing values, and
> quantised
> > > > > > formats
> > > > > > > > > pack codes with scales in ways Parquet can't describe
> today.
> > > > > > > ML-oriented
> > > > > > > > > formats like Lance are moving towards tensors,
> quantisation and
> > > > > > > indexes.
> > > > > > > > A
> > > > > > > > > type that told engines how to interpret that kind of data
> would
> > > > > > enable
> > > > > > > > new
> > > > > > > > > things. An annotation on LIST<FLOAT> doesn't.
> > > > > > > > >
> > > > > > > > > My big concern with the current focus on embeddings
> needing a
> > > > logical
> > > > > > > > type
> > > > > > > > > is compatibility. Today LIST<FLOAT> is the common format
> for
> > > > > > > embeddings,
> > > > > > > > > and Spark, BigQuery, DuckDB and pretty much every other
> library
> > > > I can
> > > > > > > > think
> > > > > > > > > of offer distance functions on plain lists, so any engine
> can
> > > > use any
> > > > > > > > file.
> > > > > > > > > If engines start requiring a VECTOR annotation, existing
> files
> > > > and
> > > > > > > > anything
> > > > > > > > > rewritten by a tool that doesn't know the type stop
> working. If
> > > > they
> > > > > > > > don't
> > > > > > > > > require it, the annotation adds nothing. We've seen this
> before
> > > > with
> > > > > > > > > strings written without the UTF8 annotation, which engines
> > > still
> > > > have
> > > > > > > to
> > > > > > > > > cope with years later.
> > > > > > > > >
> > > > > > > > > On invariants, I agree some users may want the writer to
> check
> > > > what
> > > > > > > they
> > > > > > > > > write as they write it, but that can be a writer option
> rather
> > > > than a
> > > > > > > > type.
> > > > > > > > > Readers that want evidence could get it from extra
> statistics
> > > or
> > > > > > > > metadata,
> > > > > > > > > though careful readers will still check on read.
> > > > > > > > >
> > > > > > > > > If a finite fixed-length vector contract is useful to table
> > > > formats,
> > > > > > > > > perhaps Iceberg is the place for it, with Parquet storing
> plain
> > > > > > lists.
> > > > > > > > I'd
> > > > > > > > > raise the same worry there, though: if tables start
> rejecting
> > > > > > untagged
> > > > > > > > > columns, we've just moved the split up a layer.
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > > Will
> > > > > > > > >
> > > > > > > > > On Sat, 17 Oct 2026 at 00:10, Ryan Blue <bl...@apache.org>
> > > wrote:
> > > > > > > > >
> > > > > > > > > > I agree with much of what Dan said, but I'll add some to
> this
> > > > > > thread
> > > > > > > as
> > > > > > > > > > well.
> > > > > > > > > >
> > > > > > > > > > First, I want to address the arguments that were pointed
> out:
> > > > > > > > > > 1. Other vector databases have finiteness requirements
> > > > > > > > > > 2. "Vectors are the hot stuff right now" -- let's jump
> on the
> > > > > > > bandwagon
> > > > > > > > > > 3. A general solution will take too long, finiteness will
> > > make
> > > > > > faster
> > > > > > > > > > progress
> > > > > > > > > > 4. To give a guarantee to readers that no INF/NaN/NULL
> will
> > > be
> > > > in
> > > > > > the
> > > > > > > > > > vector
> > > > > > > > > >
> > > > > > > > > > I don't think these represent our actual motivation or
> > > > position,
> > > > > > > which
> > > > > > > > > > makes them function as a straw-man. This is probably not
> what
> > > > was
> > > > > > > > > > intended, but I want to make sure that the
> misunderstanding
> > > > > > > > > > doesn't continue. For one thing, we aren't saying that we
> > > need
> > > > > > finite
> > > > > > > > > > vectors because they are "hot stuff".
> > > > > > > > > >
> > > > > > > > > > We want to build tools that use them on top of Parquet. I
> > > > think it
> > > > > > > > should
> > > > > > > > > > also be clear that we aren't advocating for finite
> elements
> > > > blindly
> > > > > > > > just
> > > > > > > > > > because other engines do (cargo cult design!). There are
> real
> > > > > > > > > > interoperability issues if we choose to go a different
> > > > direction so
> > > > > > > > this
> > > > > > > > > is
> > > > > > > > > > a "Chesterton's Fence" situation. It's important to
> > > understand
> > > > why
> > > > > > > > > > those databases
> > > > > > > > > > made that choice. I've also pointed this out as
> confirmation
> > > > that
> > > > > > > we're
> > > > > > > > > > going down a reasonable and established path.
> > > > > > > > > >
> > > > > > > > > > The third argument sounds like a simple misunderstanding.
> > > We're
> > > > > > > > pursuing
> > > > > > > > > a
> > > > > > > > > > logical type right now because that's forward compatible
> and
> > > we
> > > > > > want
> > > > > > > to
> > > > > > > > > > label the data properly and set expectations for the
> type.
> > > > Physical
> > > > > > > > type
> > > > > > > > > > changes break forward compatibility and need to be
> released
> > > in
> > > > a
> > > > > > new
> > > > > > > > > format
> > > > > > > > > > version . I think this is the source of the confusion
> since
> > > it
> > > > has
> > > > > > > been
> > > > > > > > > in
> > > > > > > > > > the same discussion, but the choice about finiteness is
> > > > separate.
> > > > > > > > > >
> > > > > > > > > > That brings us to the last argument, whether there is
> value
> > > in
> > > > > > > writers
> > > > > > > > > > participating in guarantees. I think there's value in
> adding
> > > > > > > > requirements
> > > > > > > > > > for writers. It's up to readers how they want to use
> those
> > > > > > > > requirements,
> > > > > > > > > > but a writer requirement allows a policy of "garbage in
> > > garbage
> > > > > > out"
> > > > > > > > > rather
> > > > > > > > > > than being forced to handle bad data. Why would a reader
> need
> > > > to
> > > > > > run
> > > > > > > > > > validation on every element just in case a writer had a
> bug?
> > > > And
> > > > > > why
> > > > > > > > > should
> > > > > > > > > > the potential for writer bugs mean that it has to be the
> > > > reader's
> > > > > > > > > problem?
> > > > > > > > > > If a writer produces bad data, we can absolutely reject
> it
> > > and
> > > > tell
> > > > > > > the
> > > > > > > > > > writer to fix the bug.
> > > > > > > > > >
> > > > > > > > > > I'm also puzzled by the assertion that writer guarantees
> have
> > > > > > value,
> > > > > > > > but
> > > > > > > > > > only if there are other, more specific guarantees in
> > > addition.
> > > > This
> > > > > > > > seems
> > > > > > > > > > to cede the point that a contract between readers and
> writers
> > > > makes
> > > > > > > > > sense,
> > > > > > > > > > but only if it is a _specific_ contract. But the proposed
> > > > additions
> > > > > > > > don't
> > > > > > > > > > make sense in the same way.
> > > > > > > > > >
> > > > > > > > > > Take, for example, the suggestion to also disallow
> > > zero-length
> > > > > > > vectors.
> > > > > > > > > At
> > > > > > > > > > a glance this seems similar, but I think it assumes a
> > > specific
> > > > > > > context
> > > > > > > > > and
> > > > > > > > > > is better suited to be a constraint for some use cases
> rather
> > > > than
> > > > > > > part
> > > > > > > > > of
> > > > > > > > > > what is considered valid for the type. That's because
> > > > zero-length
> > > > > > > > vectors
> > > > > > > > > > are valid vectors with a length. You can calculate
> Euclidean
> > > or
> > > > > > > > Manhattan
> > > > > > > > > > distance and use them in many contexts. They don't work
> if
> > > > you're
> > > > > > > > > > normalizing to a length of 1, but it's a choice to
> normalize
> > > > and
> > > > > > use
> > > > > > > a
> > > > > > > > > dot
> > > > > > > > > > product to calculate cosine distance. And at this point,
> > > we're
> > > > > > > talking
> > > > > > > > > > about the usefulness of restrictions for the type, not a
> > > > Platonic
> > > > > > > > ideal.
> > > > > > > > > >
> > > > > > > > > > I think the restriction that makes the most sense is that
> > > > vectors
> > > > > > > have
> > > > > > > > a
> > > > > > > > > > length and can be compared in distance calculations.
> 0-length
> > > > > > vectors
> > > > > > > > are
> > > > > > > > > > valid because they point somewhere and have a length. We
> > > > shouldn't
> > > > > > > > > restrict
> > > > > > > > > > uses to only cosine distance over normalized vectors.
> Those
> > > > are use
> > > > > > > > case
> > > > > > > > > > dependent choices and there are other valid ones. I also
> > > don't
> > > > > > think
> > > > > > > > that
> > > > > > > > > > the model used to produce a vector is part of the vector
> > > type,
> > > > just
> > > > > > > > like
> > > > > > > > > > the method used to produce a metric doesn't define its
> type:
> > > > cosine
> > > > > > > > > > similarity produces values between -1 and 1, but we
> store it
> > > > in a
> > > > > > > > > `float`,
> > > > > > > > > > not a `float(cosine)`.
> > > > > > > > > >
> > > > > > > > > > Why does it make sense to have a finiteness restriction
> as
> > > > part of
> > > > > > > the
> > > > > > > > > > type? Because that's what a (mathematical) vector is. A
> > > simple
> > > > > > search
> > > > > > > > > gives
> > > > > > > > > > a pretty common definition: "A vector is an object that
> has
> > > > both
> > > > > > > > > magnitude
> > > > > > > > > > and a direction". If a dimension/element is NaN, null, or
> > > > infinite,
> > > > > > > the
> > > > > > > > > > vector doesn't point anywhere and it doesn't have a
> > > magnitude.
> > > > Not
> > > > > > > only
> > > > > > > > > can
> > > > > > > > > > it not be used, it doesn't meet the most basic
> definition of
> > > > the
> > > > > > > type.
> > > > > > > > > >
> > > > > > > > > > Ryan
> > > > > > > > > >
> > > > > > > > > > On Thu, Oct 15, 2026 at 1:07 AM Jan Finis <
> jpfinis@gmail.com
> > > >
> > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > Daniel,
> > > > > > > > > > >
> > > > > > > > > > > alright, so no physical optimizations intended.  We
> only
> > > want
> > > > > > this
> > > > > > > > for
> > > > > > > > > > > semantic reasons, not for any significant physical
> > > > optimization.
> > > > > > If
> > > > > > > > > we're
> > > > > > > > > > > all on the same page here, that is good 👍.
> > > > > > > > > > >
> > > > > > > > > > > To phrase my remaining problem with this more
> concretely:
> > > > > > > > > > > *Let's say we add this type. What cool new stuff can
> we now
> > > > do
> > > > > > that
> > > > > > > > we
> > > > > > > > > > > couldn't do before?*
> > > > > > > > > > > That is the litmus test that a type has to pass to be
> > > useful
> > > > > > IMHO.
> > > > > > > > And
> > > > > > > > > > here
> > > > > > > > > > > I can't come up with a tangible use case that wasn't
> > > possible
> > > > > > > before.
> > > > > > > > > > > Especially if we one day have fixed size lists in
> Parquet,
> > > > the
> > > > > > > > > remaining
> > > > > > > > > > > niche for this type becomes so small, that the added
> > > > complexity
> > > > > > > isn't
> > > > > > > > > > worth
> > > > > > > > > > > the benefit IMHO.
> > > > > > > > > > >
> > > > > > > > > > > As I guess most people on here are active maintainers
> of a
> > > > > > Parquet
> > > > > > > > > > > implementation: Concretely, what would you be using
> this
> > > > type for
> > > > > > > if
> > > > > > > > we
> > > > > > > > > > had
> > > > > > > > > > > it? Would you allow math only on fields tagged with
> this
> > > > type?
> > > > > > What
> > > > > > > > if
> > > > > > > > > > that
> > > > > > > > > > > math yields INF or NaN? Not allowed?
> > > > > > > > > > >
> > > > > > > > > > > Especially if we also discuss having both VECTOR and
> > > > > > FINITE_VECTOR.
> > > > > > > > How
> > > > > > > > > > > will you guys wire this up with the type systems of
> your
> > > > engines?
> > > > > > > Do
> > > > > > > > > your
> > > > > > > > > > > engines also have two types here? Do you type-decay a
> > > > > > FINITE_VECTOR
> > > > > > > > to
> > > > > > > > > a
> > > > > > > > > > > VECTOR once someone does a multiplication that could
> yield
> > > > INF?
> > > > > > By
> > > > > > > > > > itself,
> > > > > > > > > > > a FINITE_VECTOR sounds good. Holistically looked at
> from a
> > > > type
> > > > > > > > system
> > > > > > > > > > > perspective, it comes with a lot of complexity. Do the
> > > > benefits
> > > > > > > > really
> > > > > > > > > > > outweigh the complexity here? That's why I would rather
> > > have
> > > > the
> > > > > > > bare
> > > > > > > > > > > minimum: A fixed size array (list) of floats or at
> least
> > > > just a
> > > > > > > > vector
> > > > > > > > > > > without finite. Then all these detailed questions go
> away.
> > > > > > > > > > >
> > > > > > > > > > > I'll take this one step further... Why is there any
> debate
> > > > about
> > > > > > > this
> > > > > > > > > at
> > > > > > > > > > > > all? Why can't a type simply be added to the
> > > > *specification* as
> > > > > > > an
> > > > > > > > > > > > administrative detail and be done?
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Andrew,
> > > > > > > > > > >
> > > > > > > > > > > Parquet is supposed to be a lingua franca of data. The
> > > > > > > specification
> > > > > > > > is
> > > > > > > > > > the
> > > > > > > > > > > authoritative truth for this language and all
> > > > implementations are
> > > > > > > > > > expected
> > > > > > > > > > > to implement as much as possible from this spec, so
> > > anything
> > > > in
> > > > > > the
> > > > > > > > > spec
> > > > > > > > > > > does causes implementation effort. So just adding
> anything
> > > to
> > > > > > this
> > > > > > > > spec
> > > > > > > > > > > without debate is not a good idea. It is even the
> other way
> > > > > > around:
> > > > > > > > We
> > > > > > > > > > need
> > > > > > > > > > > to have way higher rigor with the spec than we need to
> have
> > > > with
> > > > > > > any
> > > > > > > > > > > implementations. Implementations can be changed, the
> spec
> > > > needs
> > > > > > to
> > > > > > > be
> > > > > > > > > > > stable.
> > > > > > > > > > >
> > > > > > > > > > > A tag-only type that doesn't come with any
> restrictions can
> > > > be
> > > > > > > added
> > > > > > > > > with
> > > > > > > > > > > little effort. But here we want to tie restrictions to
> it
> > > (no
> > > > > > > > NaN/Inf),
> > > > > > > > > > so
> > > > > > > > > > > we would expect implementations to enforce these. So
> there
> > > is
> > > > > > some
> > > > > > > > > > > implementation effort expected here.
> > > > > > > > > > >
> > > > > > > > > > > Also, even if we only wanted a tag type, I would still
> > > argue
> > > > that
> > > > > > > you
> > > > > > > > > > > shouldn't add those without a debate. What if A wants
> XML,
> > > B
> > > > > > wants
> > > > > > > > > YAML,
> > > > > > > > > > C
> > > > > > > > > > > wants quaternions, D wants address, E wants phone
> number.
> > > If
> > > > we
> > > > > > > just
> > > > > > > > > > added
> > > > > > > > > > > any types that are "just tags", Parquet logical types
> would
> > > > just
> > > > > > > > > become a
> > > > > > > > > > > trash dump of any type that anyone ever thought to be
> > > > useful. So
> > > > > > > > > rigorous
> > > > > > > > > > > assessment of each new type candidate is necessary.
> > > > > > > > > > >
> > > > > > > > > > > Finally, many engines couple parquet with another type
> > > system
> > > > > > > (e.g.,
> > > > > > > > > > Arrow
> > > > > > > > > > > or SQL). So even with a tag only type, you at least
> still
> > > > have
> > > > > > the
> > > > > > > > > > > implementation burden of wiring this up with your
> engine's
> > > > type
> > > > > > > > system.
> > > > > > > > > > >
> > > > > > > > > > > So, just because logical types are "just" logical, that
> > > > doesn't
> > > > > > > mean
> > > > > > > > we
> > > > > > > > > > can
> > > > > > > > > > > add them without any rigor.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > A few naive questions:
> > > > > > > > > > > > 1. Would a physical definition with NANs & Inf+/-
> allow
> > > > > > efficient
> > > > > > > > > > > > encodings like ALP?
> > > > > > > > > > > > 2. If the physical representation allows values that
> make
> > > > > > > encodings
> > > > > > > > > > like
> > > > > > > > > > > > ALP impossible, can a tighter Logical type allow
> > > different
> > > > > > > > encodings?
> > > > > > > > > > > > 3. Can we RLE or binpack a vector?
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > The simple general answer here is that all these
> things do
> > > > not
> > > > > > > > require
> > > > > > > > > a
> > > > > > > > > > > logical type. Writers can already do any such
> optimization
> > > > if the
> > > > > > > > > actual
> > > > > > > > > > > data contains no NaNs & Infs. All these physical
> > > > optimizations
> > > > > > just
> > > > > > > > > > require
> > > > > > > > > > > no actual NaNs/Infs to be present in a page, they don't
> > > need
> > > > a
> > > > > > > > > guarantee
> > > > > > > > > > > that the whole file won't contain any.
> > > > > > > > > > >
> > > > > > > > > > > Cheers,
> > > > > > > > > > > Jan
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Am Do., 15. Okt. 2026 um 04:28 Uhr schrieb <
> > > > > > > kurtiscwright@gmail.com
> > > > > > > > >:
> > > > > > > > > > >
> > > > > > > > > > > > Hi Everyone,
> > > > > > > > > > > >
> > > > > > > > > > > > Glad to see a Vector type being added. I have a few
> use
> > > > cases I
> > > > > > > > want
> > > > > > > > > to
> > > > > > > > > > > > pursue once we have an agreed upon definition.
> > > > > > > > > > > >
> > > > > > > > > > > > In regards to that definition discussion. I feel
> > > > comfortable
> > > > > > with
> > > > > > > > the
> > > > > > > > > > > idea
> > > > > > > > > > > > of the semantic meaning Dan raises though I can’t
> shake a
> > > > > > slight
> > > > > > > > > > > discomfort
> > > > > > > > > > > > (maybe FOMO) with limiting values in the physical
> > > > definition.
> > > > > > > > > > > >
> > > > > > > > > > > > A few naive questions:
> > > > > > > > > > > > 1. Would a physical definition with NANs & Inf+/-
> allow
> > > > > > efficient
> > > > > > > > > > > > encodings like ALP?
> > > > > > > > > > > > 2. If the physical representation allows values that
> make
> > > > > > > encodings
> > > > > > > > > > like
> > > > > > > > > > > > ALP impossible, can a tighter Logical type allow
> > > different
> > > > > > > > encodings?
> > > > > > > > > > > > 3. Can we RLE or binpack a vector?
> > > > > > > > > > > >
> > > > > > > > > > > > Best Regards,
> > > > > > > > > > > > Kurtis C. Wright
> > > > > > > > > > > > > On Oct 14, 2026, at 12:45, Daniel Weeks <
> > > > dweeks@apache.org>
> > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > Hey Jan,
> > > > > > > > > > > > >
> > > > > > > > > > > > > There's a lot that I disagree with respect to the
> > > > framing of
> > > > > > > > these
> > > > > > > > > > > > > arguments and how we should be thinking about
> > > introducing
> > > > > > > vector
> > > > > > > > > as a
> > > > > > > > > > > > > logical type.  There's a repeated tendency to let
> the
> > > > > > physical
> > > > > > > > > > > > > representation or performance drive discussion
> around
> > > > types
> > > > > > as
> > > > > > > > > > opposed
> > > > > > > > > > > to
> > > > > > > > > > > > > what it represents.
> > > > > > > > > > > > >
> > > > > > > > > > > > > In parquet, establishing semantic meaning is the
> > > purpose
> > > > of
> > > > > > > > logical
> > > > > > > > > > > > types.
> > > > > > > > > > > > > The logical type separates how to interpret a value
> > > from
> > > > its
> > > > > > > > > physical
> > > > > > > > > > > > > representation.  We can represent a timestamp with
> > > INT64
> > > > or
> > > > > > > even
> > > > > > > > a
> > > > > > > > > > > string
> > > > > > > > > > > > > value, so why have a type for it? Even though a
> > > > > > > FIXED_SIZED_LIST
> > > > > > > > > and
> > > > > > > > > > > > VECTOR
> > > > > > > > > > > > > may have the exact same representation and
> performance,
> > > > they
> > > > > > > have
> > > > > > > > > > > > different
> > > > > > > > > > > > > semantic meaning.  Databases and engines across the
> > > > ecosystem
> > > > > > > > > already
> > > > > > > > > > > > > recognize this differentiation and have separate
> > > > > > > representations.
> > > > > > > > > > The
> > > > > > > > > > > > > choice to separate these isn't because the
> discussion
> > > is
> > > > too
> > > > > > > > > tedious;
> > > > > > > > > > > > > they're fundamentally different things.
> > > > > > > > > > > > >
> > > > > > > > > > > > > The finite requirement is what makes a vector a
> > > > reflection of
> > > > > > > > what
> > > > > > > > > it
> > > > > > > > > > > > > represents.  As we've discussed in the thread,
> > > non-finite
> > > > > > > values
> > > > > > > > in
> > > > > > > > > > the
> > > > > > > > > > > > > representation violate the properties of a vector
> and
> > > the
> > > > > > value
> > > > > > > > > > ceases
> > > > > > > > > > > to
> > > > > > > > > > > > > be a vector.  On the other hand, a 0 vector is
> still a
> > > > vector
> > > > > > > > (you
> > > > > > > > > > can
> > > > > > > > > > > > > argue the utility of it, but it is a valid vector).
> > > > > > > > > Implementations
> > > > > > > > > > > can
> > > > > > > > > > > > > choose what they validate or how to handle
> execution
> > > > errors;
> > > > > > > > that's
> > > > > > > > > > an
> > > > > > > > > > > > > implementation concern, not a requirement.
> Operations
> > > > > > > performed
> > > > > > > > > on a
> > > > > > > > > > > > > vector depend on the context in which they are
> used,
> > > but
> > > > the
> > > > > > > > > > > > representation
> > > > > > > > > > > > > doesn't need to solve scenarios where that
> application
> > > > > > doesn't
> > > > > > > > make
> > > > > > > > > > > > sense.
> > > > > > > > > > > > >
> > > > > > > > > > > > > The guarantees or context needed to use a vector
> do not
> > > > need
> > > > > > to
> > > > > > > > be
> > > > > > > > > > > > inherent
> > > > > > > > > > > > > in the type for it to be a useful type.  We don't
> need
> > > > the
> > > > > > > > currency
> > > > > > > > > > > used
> > > > > > > > > > > > > for decimal or whether a floating point is a
> normalized
> > > > value
> > > > > > > as
> > > > > > > > > part
> > > > > > > > > > > of
> > > > > > > > > > > > > the type to make use of those types.  This
> conflates
> > > the
> > > > use
> > > > > > > > cases
> > > > > > > > > > with
> > > > > > > > > > > > the
> > > > > > > > > > > > > type system and unnecessarily attempts to draw the
> > > > context
> > > > > > into
> > > > > > > > the
> > > > > > > > > > > type
> > > > > > > > > > > > > definition.  Similarly, you don't need to capture
> > > > > > > normalization,
> > > > > > > > > > model,
> > > > > > > > > > > > > etc. as part of the type for it to be useful.
> > > > > > > > > > > > >
> > > > > > > > > > > > > So the reason we want to add a vector type (or
> > > > FINITE_VECTOR)
> > > > > > > is
> > > > > > > > > > > because
> > > > > > > > > > > > we
> > > > > > > > > > > > > want to store vectors with the semantic meaning and
> > > > > > > requirements
> > > > > > > > > > based
> > > > > > > > > > > on
> > > > > > > > > > > > > what they represent.
> > > > > > > > > > > > >
> > > > > > > > > > > > > -Dan
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >> On Wed, Oct 14, 2026 at 1:22 AM Jan Finis <
> > > > > > jpfinis@gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> A bit late to the party, but here we go:
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> I have the strong sentiment that the FINITE_VECTOR
> > > type
> > > > as
> > > > > > it
> > > > > > > is
> > > > > > > > > > > > currently
> > > > > > > > > > > > >> being discussed will not be very useful and just
> > > > increase
> > > > > > > > > > > implementation
> > > > > > > > > > > > >> complexity. The arguments for the type are all
> quite
> > > > weak
> > > > > > > > (details
> > > > > > > > > > see
> > > > > > > > > > > > >> below). The FINITE part doesn't seem to yield any
> > > value
> > > > and
> > > > > > is
> > > > > > > > > not a
> > > > > > > > > > > > useful
> > > > > > > > > > > > >> restriction by itself. The vector (i.e. fixed
> length)
> > > > part
> > > > > > > > should
> > > > > > > > > > > > rather be
> > > > > > > > > > > > >> generalized instead of making a one-off thing. In
> > > > general,
> > > > > > the
> > > > > > > > > > > existing
> > > > > > > > > > > > >> types are already working well for uncompressed
> > > > vectors. The
> > > > > > > > real
> > > > > > > > > > > > challenge
> > > > > > > > > > > > >> that would bring real benefits is supporting
> > > > quantization.
> > > > > > > This
> > > > > > > > is
> > > > > > > > > > > where
> > > > > > > > > > > > >> you get orders of magnitude improvements to
> storage
> > > > space
> > > > > > and
> > > > > > > > > > > processing
> > > > > > > > > > > > >> speed.
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> Let's first discuss the FINITE part. So far, the
> only
> > > > > > > arguments
> > > > > > > > I
> > > > > > > > > > have
> > > > > > > > > > > > read
> > > > > > > > > > > > >> for this are:
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> 1. many other vector databases have it
> > > > > > > > > > > > >> 2. vectors are the hot stuff right now, so Parquet
> > > > better
> > > > > > > boast
> > > > > > > > > > having
> > > > > > > > > > > > >> "native" type support for vectors
> > > > > > > > > > > > >> 3. by focussing on this common special case, we
> can
> > > make
> > > > > > rapid
> > > > > > > > > > > progress,
> > > > > > > > > > > > >> while a general solution would take too long to
> > > specify
> > > > > > > > > > > > >> 4. to give a guarantee to readers that no
> INF/NaN/NULL
> > > > will
> > > > > > be
> > > > > > > > in
> > > > > > > > > > the
> > > > > > > > > > > > >> vector
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> the first two arguments are quite weak by
> themselves.
> > > > Vector
> > > > > > > > > > databases
> > > > > > > > > > > > are
> > > > > > > > > > > > >> very specialized engines. Such engines can of
> course
> > > be
> > > > very
> > > > > > > > > > > > opinionated on
> > > > > > > > > > > > >> their data types. A general format like Parquet
> must
> > > > instead
> > > > > > > > weigh
> > > > > > > > > > > > whether
> > > > > > > > > > > > >> a special type is worth the added complexity over
> a
> > > more
> > > > > > > > general,
> > > > > > > > > > > widely
> > > > > > > > > > > > >> applicable type. In this case, it is IMHO not at
> all.
> > > > And
> > > > > > just
> > > > > > > > > > having
> > > > > > > > > > > a
> > > > > > > > > > > > >> type called "vector" for advertisement reasons to
> make
> > > > > > Parquet
> > > > > > > > > > appear
> > > > > > > > > > > > more
> > > > > > > > > > > > >> modern seems a non-goal in the first place.
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> The third "time to market" argument is valid for
> > > systems
> > > > > > that
> > > > > > > > > don't
> > > > > > > > > > > > need a
> > > > > > > > > > > > >> stable interface. It is a very dangerous attitude
> for
> > > > > > formats
> > > > > > > > like
> > > > > > > > > > > > Parquet
> > > > > > > > > > > > >> that must be stable long term and should have a
> > > > > > consolidated,
> > > > > > > > > > concise,
> > > > > > > > > > > > >> non-redundant type system. For such long term
> storage
> > > > > > formats,
> > > > > > > > we
> > > > > > > > > > need
> > > > > > > > > > > > to
> > > > > > > > > > > > >> take the time to build the right thing, as any
> > > > shortcuts we
> > > > > > > take
> > > > > > > > > > will
> > > > > > > > > > > > stay
> > > > > > > > > > > > >> as blemishes in the format forever. If I read the
> > > > discussion
> > > > > > > > > > > correctly,
> > > > > > > > > > > > >> this is actually the main argument as of now: We
> have
> > > > found
> > > > > > > that
> > > > > > > > > it
> > > > > > > > > > is
> > > > > > > > > > > > hard
> > > > > > > > > > > > >> to align on how a fixed size list type should look
> > > > like, so
> > > > > > we
> > > > > > > > > want
> > > > > > > > > > to
> > > > > > > > > > > > go
> > > > > > > > > > > > >> the easier route of a specialized type. This seems
> > > like
> > > > a
> > > > > > > > > > > > >> dangerous shortcut fueled by feature recency
> bias. Ask
> > > > > > > yourself:
> > > > > > > > > > Would
> > > > > > > > > > > > we
> > > > > > > > > > > > >> want this, if Parquet already had a fixed size
> list
> > > > type?
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> So finally, the only technical argument is the
> fourth
> > > > one:
> > > > > > "It
> > > > > > > > > > signals
> > > > > > > > > > > > >> semantics and ensures that no bad data lands in
> the
> > > > > > format". I
> > > > > > > > see
> > > > > > > > > > two
> > > > > > > > > > > > big
> > > > > > > > > > > > >> problems with this:
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> 1. Readers *still* need to test for NaN / INF
> because
> > > > > > Parquet
> > > > > > > > > files
> > > > > > > > > > > may
> > > > > > > > > > > > be
> > > > > > > > > > > > >> corrupt. Nothing can stop a malicious or buggy
> writer
> > > > from
> > > > > > > > > > physically
> > > > > > > > > > > > >> writing NaN or INF. *So you save no single line of
> > > code
> > > > in a
> > > > > > > > > reader.
> > > > > > > > > > > You
> > > > > > > > > > > > >> save no branches, as you still need to handle the
> > > > INF/NaN
> > > > > > > case.*
> > > > > > > > > So
> > > > > > > > > > > this
> > > > > > > > > > > > >> doesn't help with performance at all. The only
> > > > difference
> > > > > > is,
> > > > > > > > that
> > > > > > > > > > the
> > > > > > > > > > > > >> error message changes from "found an invalid INF
> value
> > > > in
> > > > > > the
> > > > > > > > > data"
> > > > > > > > > > to
> > > > > > > > > > > > >> "Your parquet file seems to be corrupt". It is
> > > actually
> > > > > > quite
> > > > > > > > the
> > > > > > > > > > > > opposite,
> > > > > > > > > > > > >> readers and writers get more complex having to
> handle
> > > > this
> > > > > > new
> > > > > > > > > type.
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> If you now argue that it's not about performance,
> it's
> > > > about
> > > > > > > > > defined
> > > > > > > > > > > > >> semantics and erroring out early in writers, then
> the
> > > > second
> > > > > > > > > problem
> > > > > > > > > > > > >> arises:
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> 2. The proposed no-NaN/INF/NULL guarantee is just
> one
> > > of
> > > > > > > > multiple
> > > > > > > > > > > > >> guarantees that you need for sensible vector
> > > > processing. If
> > > > > > > "no
> > > > > > > > > > > > >> INF/NaN/NULL" was the only guarantee you needed, I
> > > would
> > > > > > agree
> > > > > > > > > that
> > > > > > > > > > > > >> FINITE_VECTOR is useful, but it is by no means the
> > > only
> > > > > > > > guarantee
> > > > > > > > > > you
> > > > > > > > > > > > need,
> > > > > > > > > > > > >> instead, depending on the distance metric, you
> also
> > > > need the
> > > > > > > > > > following
> > > > > > > > > > > > >> guarantees. E.g.:
> > > > > > > > > > > > >>
> > > > > > > > > > > > >>   - No vector may be the 0 vector, otherwise,
> e.g.,
> > > > cosine
> > > > > > > > > > similarity
> > > > > > > > > > > is
> > > > > > > > > > > > >>   undefined. Cosine similarity is often used as
> > > > default, so
> > > > > > > this
> > > > > > > > > is
> > > > > > > > > > > > >> actually
> > > > > > > > > > > > >>   quite important in practice.
> > > > > > > > > > > > >>   - If we want to use a dot product, then all
> vectors
> > > > need
> > > > > > to
> > > > > > > > > > > guarantee
> > > > > > > > > > > > a
> > > > > > > > > > > > >>   length of 1. This would actually be quite
> useful for
> > > > > > > > performance
> > > > > > > > > > > > >> reasons.
> > > > > > > > > > > > >>   - The vectors must come from the same model as
> the
> > > > vector
> > > > > > > used
> > > > > > > > > for
> > > > > > > > > > > the
> > > > > > > > > > > > >>   search. Otherwise, the search yields a numeric
> > > > result, but
> > > > > > > > that
> > > > > > > > > > > result
> > > > > > > > > > > > >> has
> > > > > > > > > > > > >>   no sensible meaning.
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> So, what use is a type that just guarantees part
> of
> > > the
> > > > > > things
> > > > > > > > you
> > > > > > > > > > > need
> > > > > > > > > > > > for
> > > > > > > > > > > > >> vector search but not others? You still need
> > > additional
> > > > > > > > guarantees
> > > > > > > > > > in
> > > > > > > > > > > > >> addition to that type. What makes INF/NaN so
> important
> > > > in
> > > > > > > > > comparison
> > > > > > > > > > > to,
> > > > > > > > > > > > >> say, "no 0 vector present", that it warrants an
> own
> > > > type,
> > > > > > > while
> > > > > > > > > the
> > > > > > > > > > > > other
> > > > > > > > > > > > >> guarantees do not? If we really want this type as
> a
> > > > basis
> > > > > > for
> > > > > > > > > > > > cross-engine
> > > > > > > > > > > > >> sharing of vectors, then it needs to give *all*
> > > > guarantees.
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> I'm not against a specialized vector type if it
> really
> > > > > > carries
> > > > > > > > > *all*
> > > > > > > > > > > > >> information to be useful by itself. If it carried
> all
> > > > that
> > > > > > > > > > > information,
> > > > > > > > > > > > >> then a reading engine could indeed validate that
> only
> > > > > > > meaningful
> > > > > > > > > > > vector
> > > > > > > > > > > > >> searches are executed. But as long as it is just a
> > > > > > half-baked
> > > > > > > > > thing,
> > > > > > > > > > > it
> > > > > > > > > > > > is
> > > > > > > > > > > > >> not useful by itself and just a burden on each
> > > > implementer
> > > > > > and
> > > > > > > > > > > > maintainer.
> > > > > > > > > > > > >> If we need extra information anyway, why not just
> > > stick
> > > > to a
> > > > > > > > > general
> > > > > > > > > > > > type
> > > > > > > > > > > > >> like VECTOR (without FINITE).
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> If you're a proponent of FINITE_VECTOR, ask
> yourself
> > > (or
> > > > > > tell
> > > > > > > > us)
> > > > > > > > > > what
> > > > > > > > > > > > >> advantage you will *really* get over just a
> > > (potentially
> > > > > > fixed
> > > > > > > > > size)
> > > > > > > > > > > > List
> > > > > > > > > > > > >> of Float? As discussed, it's not performance. It's
> > > also
> > > > not
> > > > > > a
> > > > > > > > > > semantic
> > > > > > > > > > > > >> marker that is sufficient by itself. It's just a
> small
> > > > hint
> > > > > > > that
> > > > > > > > > > > allows
> > > > > > > > > > > > >> writers to error out once they see a NaN or INF
> in the
> > > > > > input,
> > > > > > > > > while
> > > > > > > > > > > they
> > > > > > > > > > > > >> will still happily write a bogus 0 vector. This
> seems
> > > > to be
> > > > > > of
> > > > > > > > > zero
> > > > > > > > > > > > >> benefit.
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> Anecdotally, we currently handle petabytes of
> vectors
> > > in
> > > > > > > > > production
> > > > > > > > > > in
> > > > > > > > > > > > >> Data360. We simply store them in a List of Float,
> with
> > > > an
> > > > > > > > > optimized
> > > > > > > > > > > > R-Level
> > > > > > > > > > > > >> reader implementation that rapidly checks that all
> > > > vectors
> > > > > > are
> > > > > > > > the
> > > > > > > > > > > same
> > > > > > > > > > > > >> length. This solution works just fine. The
> overhead of
> > > > > > R-level
> > > > > > > > > > > decoding
> > > > > > > > > > > > is
> > > > > > > > > > > > >> barely measurable. The check is super fast and
> > > R-levels
> > > > are
> > > > > > > tiny
> > > > > > > > > in
> > > > > > > > > > > > >> comparison to the payload vectors, so also the
> added
> > > > I/O is
> > > > > > > > > > > negligible.
> > > > > > > > > > > > >> This is also why I'm so firm in my sentiment: We
> are
> > > > > > running a
> > > > > > > > > > > > production
> > > > > > > > > > > > >> grade system processing trillions of vectors
> right now
> > > > and
> > > > > > see
> > > > > > > > no
> > > > > > > > > > big
> > > > > > > > > > > > >> issues with the types that Parquet currently
> provides.
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> As I don't just want to be a naysayer and point
> out
> > > > problems
> > > > > > > > with
> > > > > > > > > > the
> > > > > > > > > > > > >> current design, here are the directions that I
> would
> > > > propose
> > > > > > > > > indeed:
> > > > > > > > > > > > >>
> > > > > > > > > > > > >>   - a general fixed size list type. Don't do
> shortcuts
> > > > just
> > > > > > > > > because
> > > > > > > > > > > the
> > > > > > > > > > > > >>   discussion is too tedious.
> > > > > > > > > > > > >>   - a vector type that indeed carries enough
> > > > information to
> > > > > > be
> > > > > > > > > > really
> > > > > > > > > > > > >>   useful for vector search by itself. (e.g., model
> > > name
> > > > /
> > > > > > > > version,
> > > > > > > > > > > > >> optional
> > > > > > > > > > > > >>   guarantees for no 0 vector / normalized vector
> > > > length).
> > > > > > With
> > > > > > > > > such
> > > > > > > > > > a
> > > > > > > > > > > > >> type,
> > > > > > > > > > > > >>   you could really do well defined vector searches
> > > even
> > > > if
> > > > > > the
> > > > > > > > > > writing
> > > > > > > > > > > > >> engine
> > > > > > > > > > > > >>   is completely different to the reading one.
> > > > > > > > > > > > >>   - Standardize quantizations. This will be a real
> > > game
> > > > > > > changer
> > > > > > > > > for
> > > > > > > > > > > > >>   performance.
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> Cheers,
> > > > > > > > > > > > >> Jan
> > > > > > > > > > > > >>
> > > > > > > > > > > > >>
> > > > > > > > > > > > >>
> > > > > > > > > > > > >>
> > > > > > > > > > > > >>
> > > > > > > > > > > > >>
> > > > > > > > > > > > >>> Am Mi., 14. Okt. 2026 um 03:36 Uhr schrieb Gang
> Wu <
> > > > > > > > > > ustcwg@gmail.com
> > > > > > > > > > > >:
> > > > > > > > > > > > >>>
> > > > > > > > > > > > >>> I think separating them into FINITE_VECTOR and
> > > > > > > > FIXED_LENGTH_LIST
> > > > > > > > > > is a
> > > > > > > > > > > > >> good
> > > > > > > > > > > > >>> idea, though I think renaming FINITE_VECTOR to
> VECTOR
> > > > looks
> > > > > > > > more
> > > > > > > > > > > > natural.
> > > > > > > > > > > > >>> Should we move forward with proposing
> > > > FINITE_VECTOR/VECTOR
> > > > > > > > first?
> > > > > > > > > > > > >>>
> > > > > > > > > > > > >>> On Tue, Oct 13, 2026 at 6:26 PM Alkis
> Evlogimenos via
> > > > dev <
> > > > > > > > > > > > >>> dev@parquet.apache.org> wrote:
> > > > > > > > > > > > >>>
> > > > > > > > > > > > >>>> I suggest we go with VECTOR for the finite
> vector to
> > > > keep
> > > > > > > the
> > > > > > > > > name
> > > > > > > > > > > the
> > > > > > > > > > > > >>> same
> > > > > > > > > > > > >>>> as other engines/type systems. If we want a
> > > > non-finite one
> > > > > > > in
> > > > > > > > > the
> > > > > > > > > > > > >> future
> > > > > > > > > > > > >>> we
> > > > > > > > > > > > >>>> can name that something else or we add fixed
> size
> > > list
> > > > > > that
> > > > > > > > will
> > > > > > > > > > > serve
> > > > > > > > > > > > >>> the
> > > > > > > > > > > > >>>> same purpose without a special name.
> > > > > > > > > > > > >>>>
> > > > > > > > > > > > >>>>
> > > > > > > > > > > > >>>> On Thu, Oct 8, 2026 at 10:20 PM Ryan Blue <
> > > > > > blue@apache.org>
> > > > > > > > > > wrote:
> > > > > > > > > > > > >>>>
> > > > > > > > > > > > >>>>> A logical type would:
> > > > > > > > > > > > >>>>> - Describe data for users and readers to
> understand
> > > > what
> > > > > > > the
> > > > > > > > > file
> > > > > > > > > > > > >>>> contains
> > > > > > > > > > > > >>>>> - Signal that repetition and definition levels
> will
> > > > > > > reflect a
> > > > > > > > > > fixed
> > > > > > > > > > > > >>>> number
> > > > > > > > > > > > >>>>> of elements, which implementations can use to
> > > > optimize
> > > > > > > > decoding
> > > > > > > > > > > > >>>>> - Guarantee that there are no null elements
> > > (because
> > > > the
> > > > > > > type
> > > > > > > > > > > cannot
> > > > > > > > > > > > >> be
> > > > > > > > > > > > >>>>> applied to a list with optional elements)
> > > > > > > > > > > > >>>>> - [FINITE_VECTOR] Guarantee that no elements
> are
> > > NaN
> > > > or
> > > > > > > > > Infinity
> > > > > > > > > > > > >> values
> > > > > > > > > > > > >>>> so
> > > > > > > > > > > > >>>>> that the vectors have a finite length and are
> > > > suitable
> > > > > > for
> > > > > > > > > > distance
> > > > > > > > > > > > >>>>> calculations
> > > > > > > > > > > > >>>>>
> > > > > > > > > > > > >>>>> The finite value restriction is a restriction
> on
> > > > writers
> > > > > > to
> > > > > > > > > > prevent
> > > > > > > > > > > > >> bad
> > > > > > > > > > > > >>>>> data from being written.
> > > > > > > > > > > > >>>>>
> > > > > > > > > > > > >>>>> Ignoring logical types that are not understood
> is a
> > > > rule
> > > > > > > for
> > > > > > > > > > > > >> readers. A
> > > > > > > > > > > > >>>>> reader can handle vectors and pass it on to
> another
> > > > > > layer.
> > > > > > > > For
> > > > > > > > > > > > >>> instance,
> > > > > > > > > > > > >>>> a
> > > > > > > > > > > > >>>>> Parquet reader that doesn't support geometry
> passes
> > > > > > values
> > > > > > > as
> > > > > > > > > > > > >> `binary`.
> > > > > > > > > > > > >>>>> That binary value can be passed to a geometry
> UDF,
> > > > even
> > > > > > if
> > > > > > > > the
> > > > > > > > > > > > >>> underlying
> > > > > > > > > > > > >>>>> engine and Parquet reader doesn't have support.
> > > > > > > > > > > > >>>>>
> > > > > > > > > > > > >>>>> A tool that doesn't understand FINITE_VECTOR
> can
> > > > > > similarly
> > > > > > > > > pass a
> > > > > > > > > > > > >> list
> > > > > > > > > > > > >>> of
> > > > > > > > > > > > >>>>> floats along to a UDF. But it is still
> important to
> > > > the
> > > > > > > user
> > > > > > > > > that
> > > > > > > > > > > bad
> > > > > > > > > > > > >>>>> values are not present in the data. In your
> > > example,
> > > > bad
> > > > > > > > values
> > > > > > > > > > are
> > > > > > > > > > > > >>>> vectors
> > > > > > > > > > > > >>>>> that cause the centroid calculation to fail,
> like
> > > > vectors
> > > > > > > > with
> > > > > > > > > > > > >> infinite
> > > > > > > > > > > > >>>>> length.
> > > > > > > > > > > > >>>>>
> > > > > > > > > > > > >>>>> Ryan
> > > > > > > > > > > > >>>>>
> > > > > > > > > > > > >>>>> On Thu, Oct 8, 2026 at 12:54 PM Will Edwards
> via
> > > dev
> > > > <
> > > > > > > > > > > > >>>>> dev@parquet.apache.org>
> > > > > > > > > > > > >>>>> wrote:
> > > > > > > > > > > > >>>>>
> > > > > > > > > > > > >>>>>> Could we clarify the use cases for a VECTOR
> > > > annotation
> > > > > > on
> > > > > > > > > > > > >>> LIST<FLOAT>?
> > > > > > > > > > > > >>>>>> Specifically, what does it make possible
> that's
> > > > > > impossible
> > > > > > > > > > today,
> > > > > > > > > > > > >> and
> > > > > > > > > > > > >>>>> what
> > > > > > > > > > > > >>>>>> does it make easier that's currently hard?
> > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > >>>>>> I'm struggling to find a strong answer. I'd
> like
> > > > every
> > > > > > > > engine
> > > > > > > > > to
> > > > > > > > > > > > >>> offer
> > > > > > > > > > > > >>>>>> vector functions on lists of floats without
> > > needing
> > > > a
> > > > > > new
> > > > > > > > > > logical
> > > > > > > > > > > > >>> type.
> > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > >>>>>> To be clear, I'm still enthusiastic about a
> vector
> > > > type
> > > > > > > that
> > > > > > > > > > > > >>> describes
> > > > > > > > > > > > >>>>> how
> > > > > > > > > > > > >>>>>> to unpack and scale composite representations
> > > > (quantised
> > > > > > > > codes
> > > > > > > > > > > plus
> > > > > > > > > > > > >>>>> scales,
> > > > > > > > > > > > >>>>>> codebooks, etc.), in the direction ML is
> going.
> > > That
> > > > > > adds
> > > > > > > > > > > > >> information
> > > > > > > > > > > > >>>>>> engines don't have. A label on LIST<FLOAT>
> > > doesn't:
> > > > > > > engines
> > > > > > > > > > > already
> > > > > > > > > > > > >>>>>> understand the physical type, and many already
> > > offer
> > > > > > > vector
> > > > > > > > > > > > >> functions
> > > > > > > > > > > > >>>> on
> > > > > > > > > > > > >>>>>> it. I don't see a performance benefit, and
> > > > NaN/null/Inf
> > > > > > > > checks
> > > > > > > > > > are
> > > > > > > > > > > > >>>> cheap
> > > > > > > > > > > > >>>>>> enough that engines should do them anyway. The
> > > label
> > > > > > also
> > > > > > > > > > doesn't
> > > > > > > > > > > > >> say
> > > > > > > > > > > > >>>>> what
> > > > > > > > > > > > >>>>>> you can do with the column. Knowing that a
> column
> > > > holds
> > > > > > > > > > embeddings
> > > > > > > > > > > > >>>>> doesn't
> > > > > > > > > > > > >>>>>> say how they can be used, just as zipcode and
> age
> > > > are
> > > > > > both
> > > > > > > > > > > integers
> > > > > > > > > > > > >>> but
> > > > > > > > > > > > >>>>>> multiplying them is meaningless.
> > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > >>>>>> My concern is the opposite effect. We're
> > > > establishing
> > > > > > that
> > > > > > > > > > engines
> > > > > > > > > > > > >>>> should
> > > > > > > > > > > > >>>>>> ignore logical types they don't understand.
> So a
> > > > tool
> > > > > > that
> > > > > > > > > > doesn't
> > > > > > > > > > > > >>> know
> > > > > > > > > > > > >>>>>> VECTOR will read the column as LIST<FLOAT>,
> let
> > > the
> > > > user
> > > > > > > > > compute
> > > > > > > > > > > > >>>>> centroids
> > > > > > > > > > > > >>>>>> (say), and write plain LIST<FLOAT> back out.
> If
> > > > other
> > > > > > > > engines
> > > > > > > > > > > > >>> restrict
> > > > > > > > > > > > >>>>>> embedding functions to VECTOR-annotated
> columns,
> > > > that
> > > > > > file
> > > > > > > > > stops
> > > > > > > > > > > > >>>> working
> > > > > > > > > > > > >>>>>> with them, through no fault of the user. An
> > > > annotation
> > > > > > > that
> > > > > > > > > > > doesn't
> > > > > > > > > > > > >>>>> change
> > > > > > > > > > > > >>>>>> what the data is shouldn't be able to make it
> > > > unusable.
> > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > >>>>>> Am I missing a use case?
> > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > >>>>>> Best, Will
> > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > >>>>>> On Thu, 8 Oct 2026 at 19:53, Steven Wu <
> > > > > > > > stevenz3wu@gmail.com>
> > > > > > > > > > > > >> wrote:
> > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > >>>>>>> I also think a FINITE_VECTOR type brings a
> lot of
> > > > > > values
> > > > > > > to
> > > > > > > > > > many
> > > > > > > > > > > > >>> use
> > > > > > > > > > > > >>>>>> cases.
> > > > > > > > > > > > >>>>>>> It is would be great to make progress on the
> > > > > > > FINITE_VECTOR,
> > > > > > > > > and
> > > > > > > > > > > > >> not
> > > > > > > > > > > > >>>> to
> > > > > > > > > > > > >>>>>> fit
> > > > > > > > > > > > >>>>>>> every use cases into the FIXED_LENGTH_LIST.
> > > > > > > > > > > > >>>>>>>
> > > > > > > > > > > > >>>>>>> On Mon, Oct 5, 2026 at 1:26 PM Ryan Blue <
> > > > > > > blue@apache.org>
> > > > > > > > > > > > >> wrote:
> > > > > > > > > > > > >>>>>>>
> > > > > > > > > > > > >>>>>>>> Going through this thread again, it looks
> like
> > > > there
> > > > > > > are a
> > > > > > > > > few
> > > > > > > > > > > > >>>> things
> > > > > > > > > > > > >>>>>>> that
> > > > > > > > > > > > >>>>>>>> people generally agree on:
> > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > >>>>>>>> - In the short term, we want to introduce a
> > > > logical
> > > > > > type
> > > > > > > > or
> > > > > > > > > > > > >>> types;
> > > > > > > > > > > > >>>> a
> > > > > > > > > > > > >>>>>>> better
> > > > > > > > > > > > >>>>>>>> physical representation can be done later.
> We
> > > can
> > > > > > focus
> > > > > > > on
> > > > > > > > > > > > >>> logical
> > > > > > > > > > > > >>>>>> types.
> > > > > > > > > > > > >>>>>>>> - There is precedent in logical types for
> > > > restricting
> > > > > > > > > values,
> > > > > > > > > > > > >>> like
> > > > > > > > > > > > >>>>> UTF8
> > > > > > > > > > > > >>>>>>>> strings to contain only UTF-8 codepoints. I
> > > think
> > > > we
> > > > > > > also
> > > > > > > > > > agree
> > > > > > > > > > > > >>>> that
> > > > > > > > > > > > >>>>> we
> > > > > > > > > > > > >>>>>>>> could restrict vector elements to
> non-nested or
> > > > > > > primitive
> > > > > > > > > > > > >> types.
> > > > > > > > > > > > >>>>>>>> - There are many examples of databases that
> > > > restrict
> > > > > > > their
> > > > > > > > > > > > >> vector
> > > > > > > > > > > > >>>>> types
> > > > > > > > > > > > >>>>>>> to
> > > > > > > > > > > > >>>>>>>> non-null, finite elements.
> > > > > > > > > > > > >>>>>>>> - Some people want to have write-time
> guarantees
> > > > that
> > > > > > > > > vectors
> > > > > > > > > > > > >>> have
> > > > > > > > > > > > >>>>> only
> > > > > > > > > > > > >>>>>>>> non-null, finite elements
> > > > > > > > > > > > >>>>>>>> - Some people want to extend this work to
> > > include
> > > > > > > tensors,
> > > > > > > > > > > > >>> shorter
> > > > > > > > > > > > >>>>>>> floating
> > > > > > > > > > > > >>>>>>>> point representations, variable lengths,
> NaN or
> > > > > > Infinity
> > > > > > > > > > > > >>> elements,
> > > > > > > > > > > > >>>>> etc.
> > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > >>>>>>>> It sounds like we are nearly there, but
> have two
> > > > > > halves
> > > > > > > of
> > > > > > > > > the
> > > > > > > > > > > > >>>>>> community
> > > > > > > > > > > > >>>>>>>> that care about different things. That just
> > > means
> > > > we
> > > > > > > > should
> > > > > > > > > > > > >> have
> > > > > > > > > > > > >>>>>> multiple
> > > > > > > > > > > > >>>>>>>> types. It's entirely reasonable to want a
> type
> > > > that
> > > > > > > allows
> > > > > > > > > NaN
> > > > > > > > > > > > >>>>> values,
> > > > > > > > > > > > >>>>>>> and
> > > > > > > > > > > > >>>>>>>> it's also reasonable for those of us
> building
> > > > > > databases,
> > > > > > > > > table
> > > > > > > > > > > > >>>>>> standards,
> > > > > > > > > > > > >>>>>>>> and vector indexes to care about guarantees
> that
> > > > > > > elements
> > > > > > > > > are
> > > > > > > > > > > > >>>> finite.
> > > > > > > > > > > > >>>>>>>> There's more than enough examples to show
> that
> > > > this
> > > > > > is a
> > > > > > > > > > > > >>> legitimate
> > > > > > > > > > > > >>>>> use
> > > > > > > > > > > > >>>>>>>> case.
> > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > >>>>>>>> Since I don't see the concerns of the
> different
> > > > use
> > > > > > > cases
> > > > > > > > > > > > >>> changing,
> > > > > > > > > > > > >>>>> it
> > > > > > > > > > > > >>>>>>>> looks like we need to find a compromise to
> move
> > > > > > > forward. I
> > > > > > > > > > > > >>> propose
> > > > > > > > > > > > >>>>> that
> > > > > > > > > > > > >>>>>>> we
> > > > > > > > > > > > >>>>>>>> introduce a FINITE_VECTOR type and
> separately a
> > > > > > > > > > > > >> FIXED_LENGTH_LIST
> > > > > > > > > > > > >>>> or
> > > > > > > > > > > > >>>>>>>> another VECTOR type that allows infinite
> and NaN
> > > > > > values.
> > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > >>>>>>>> Can we agree on a compromise? Is there a
> strong
> > > > reason
> > > > > > > to
> > > > > > > > > try
> > > > > > > > > > > > >> to
> > > > > > > > > > > > >>>> fit
> > > > > > > > > > > > >>>>>>> these
> > > > > > > > > > > > >>>>>>>> use cases into a single type? What if that
> means
> > > > not
> > > > > > > > making
> > > > > > > > > > > > >>>> progress?
> > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > >>>>>>>> Ryan
> > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > >>>>>>>> On Mon, Oct 5, 2026 at 12:15 PM Philipp
> > > Fischbeck
> > > > via
> > > > > > > dev
> > > > > > > > <
> > > > > > > > > > > > >>>>>>>> dev@parquet.apache.org> wrote:
> > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > >>>>>>>>> Thinking about this more, I am also fine
> with
> > > > pulling
> > > > > > > out
> > > > > > > > > the
> > > > > > > > > > > > >>>>>> “NaN/Inf”
> > > > > > > > > > > > >>>>>>>>> requirement. Either into a type boolean (my
> > > > > > > preference),
> > > > > > > > or
> > > > > > > > > > > > >>>> putting
> > > > > > > > > > > > >>>>>> no
> > > > > > > > > > > > >>>>>>>>> restrictions on the Parquet level,
> allowing all
> > > > IEEE
> > > > > > > 754
> > > > > > > > > > > > >> float
> > > > > > > > > > > > >>>>>> numbers.
> > > > > > > > > > > > >>>>>>>> If
> > > > > > > > > > > > >>>>>>>>> some top-level application wants to put
> > > > additional
> > > > > > > > > > > > >> constraints
> > > > > > > > > > > > >>> on
> > > > > > > > > > > > >>>>> the
> > > > > > > > > > > > >>>>>>>> data
> > > > > > > > > > > > >>>>>>>>> type, like Iceberg not allowing
> NaN/infinity,
> > > it
> > > > can
> > > > > > > > still
> > > > > > > > > do
> > > > > > > > > > > > >>> so
> > > > > > > > > > > > >>>>> one
> > > > > > > > > > > > >>>>>>>> layer
> > > > > > > > > > > > >>>>>>>>> above. It seems like this was the biggest
> point
> > > > of
> > > > > > > > > discussion
> > > > > > > > > > > > >>>>>>> remaining.
> > > > > > > > > > > > >>>>>>>>> All these options work for me, and we
> should
> > > move
> > > > > > > forward
> > > > > > > > > > > > >> with
> > > > > > > > > > > > >>>> this
> > > > > > > > > > > > >>>>>>>>> proposal.
> > > > > > > > > > > > >>>>>>>>>
> > > > > > > > > > > > >>>>>>>>> Concerning additional element types like
> bf16,
> > > > they
> > > > > > can
> > > > > > > > be
> > > > > > > > > > > > >>>>> introduced
> > > > > > > > > > > > >>>>>>> as
> > > > > > > > > > > > >>>>>>>>> Parquet types (at least in the current
> > > LIST-based
> > > > > > spec
> > > > > > > > > > > > >>> proposal).
> > > > > > > > > > > > >>>>>>>>>
> > > > > > > > > > > > >>>>>>>>> Concerning Will’s suggestion around
> > > representing
> > > > > > VECTOR
> > > > > > > > as
> > > > > > > > > a
> > > > > > > > > > > > >>>>> variable
> > > > > > > > > > > > >>>>>>>>> length list with a MIN/MAX_LENGTH metadata
> > > > > > > restriction: I
> > > > > > > > > > > > >> think
> > > > > > > > > > > > >>>>> that
> > > > > > > > > > > > >>>>>> a
> > > > > > > > > > > > >>>>>>>>> dedicated logical VECTOR type still
> provides
> > > > enough
> > > > > > > > > benefits
> > > > > > > > > > > > >> by
> > > > > > > > > > > > >>>>>>>> describing
> > > > > > > > > > > > >>>>>>>>> the semantics of the data, and allowing to
> > > > capture
> > > > > > > future
> > > > > > > > > > > > >>> changes
> > > > > > > > > > > > >>>>>>> around
> > > > > > > > > > > > >>>>>>>>> statistics. In that case, the length is
> better
> > > > > > captured
> > > > > > > > as
> > > > > > > > > an
> > > > > > > > > > > > >>>> exact
> > > > > > > > > > > > >>>>>>>> length
> > > > > > > > > > > > >>>>>>>>> for this type specifically. I’m also not
> sure
> > > > about
> > > > > > > other
> > > > > > > > > use
> > > > > > > > > > > > >>>> cases
> > > > > > > > > > > > >>>>>> for
> > > > > > > > > > > > >>>>>>>>> MIN_LENGTH, so introducing a new type
> > > constraint
> > > > > > system
> > > > > > > > > just
> > > > > > > > > > > > >>> for
> > > > > > > > > > > > >>>>> the
> > > > > > > > > > > > >>>>>>>>> MIN_LENGTH=MAX_LENGTH case for vectors
> doesn’t
> > > > have
> > > > > > > > enough
> > > > > > > > > > > > >>> value
> > > > > > > > > > > > >>>> in
> > > > > > > > > > > > >>>>>> my
> > > > > > > > > > > > >>>>>>>>> view.
> > > > > > > > > > > > >>>>>>>>>
> > > > > > > > > > > > >>>>>>>>> Best,
> > > > > > > > > > > > >>>>>>>>> Philipp
> > > > > > > > > > > > >>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>
> > > > > > > > > > > > >>>>>>>>> On Thu, Oct 1, 2026 at 9:00 AM Will
> Edwards via
> > > > dev <
> > > > > > > > > > > > >>>>>>>>> dev@parquet.apache.org>
> > > > > > > > > > > > >>>>>>>>> wrote:
> > > > > > > > > > > > >>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>> +1 Sebastian, a vector type suggests usage
> > > > beyond
> > > > > > > > > > > > >> normalized
> > > > > > > > > > > > >>>>>>> embeddings
> > > > > > > > > > > > >>>>>>>>> and
> > > > > > > > > > > > >>>>>>>>>> engines really aren't going to be able to
> > > elide
> > > > all
> > > > > > > the
> > > > > > > > > > > > >>> checks
> > > > > > > > > > > > >>>>>>> anyway.
> > > > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>> Antoine, you suggested we explore metadata
> > > > solutions
> > > > > > > too
> > > > > > > > > :)
> > > > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>> Today, a database supporting CHAR(N) or
> > > > VARCHAR(n)
> > > > > > > can't
> > > > > > > > > > > > >>>> express
> > > > > > > > > > > > >>>>>> this
> > > > > > > > > > > > >>>>>>>> in
> > > > > > > > > > > > >>>>>>>>>> Parquet.  Engines can write opaque blobs
> they
> > > > > > > understand
> > > > > > > > > in
> > > > > > > > > > > > >>> the
> > > > > > > > > > > > >>>>>>>> metadata
> > > > > > > > > > > > >>>>>>>>>> (e.g. Arrow does) so the extra type info
> > > > > > round-trips,
> > > > > > > > but
> > > > > > > > > > > > >>> this
> > > > > > > > > > > > >>>>>> isn't
> > > > > > > > > > > > >>>>>>>>>> standardized, portable, nor interoperable.
> > > > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>> So perhaps standardizing constraint
> metadata
> > > or
> > > > new
> > > > > > > > Thrift
> > > > > > > > > > > > >>>> fields
> > > > > > > > > > > > >>>>>> (my
> > > > > > > > > > > > >>>>>>>>> pref)
> > > > > > > > > > > > >>>>>>>>>> can generically support VECTOR(N) whilst
> also
> > > > being
> > > > > > > more
> > > > > > > > > > > > >>>> broadly
> > > > > > > > > > > > >>>>>>> useful
> > > > > > > > > > > > >>>>>>>>> for
> > > > > > > > > > > > >>>>>>>>>> other use cases and data types?
> > > > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>> Imagine metadata or Thrift fields to tag
> the
> > > > > > > MIN_LENGTH
> > > > > > > > > and
> > > > > > > > > > > > >>>>>>> MAX_LENGTH
> > > > > > > > > > > > >>>>>>>>> for
> > > > > > > > > > > > >>>>>>>>>> Parquet's variable-length physcial types.
> > > These
> > > > > > would
> > > > > > > > > > > > >> count
> > > > > > > > > > > > >>>>>> logical
> > > > > > > > > > > > >>>>>>>>> units
> > > > > > > > > > > > >>>>>>>>>> (e.g. UTF8 code points for STRING) rather
> than
> > > > > > bytes.
> > > > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>> Then one engine's VECTOR(N) can be
> written to
> > > > > > Parquet
> > > > > > > as
> > > > > > > > > > > > >>>> LIST(N)
> > > > > > > > > > > > >>>>>> with
> > > > > > > > > > > > >>>>>>>>>> MIN_LENGTH = MAX_LENGTH = N and then
> readers -
> > > > even
> > > > > > > > > > > > >> different
> > > > > > > > > > > > >>>>>>> engines -
> > > > > > > > > > > > >>>>>>>>> can
> > > > > > > > > > > > >>>>>>>>>> pick up these constraints if they support
> > > doing
> > > > so.
> > > > > > > We
> > > > > > > > > > > > >> don't
> > > > > > > > > > > > >>>>>>> actually
> > > > > > > > > > > > >>>>>>>>> need
> > > > > > > > > > > > >>>>>>>>>> a logical type to express a VECTOR then.
> > > > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>> This also allows standard SQL's CHAR(N)
> and
> > > > > > VARCHAR(N)
> > > > > > > > to
> > > > > > > > > > > > >> be
> > > > > > > > > > > > >>>>>>> expressed
> > > > > > > > > > > > >>>>>>>> on
> > > > > > > > > > > > >>>>>>>>>> UTF8 columns etc.
> > > > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>> This also seems generally useful in the
> future
> > > > as we
> > > > > > > > > > > > >> approach
> > > > > > > > > > > > >>>>>>>> expressing
> > > > > > > > > > > > >>>>>>>>>> other tensor data shapes and types.
> > > > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>> This won't change the mechanics: the
> storage
> > > > type
> > > > > > > > doesn't
> > > > > > > > > > > > >>>> enforce
> > > > > > > > > > > > >>>>>>> these
> > > > > > > > > > > > >>>>>>>>>> constraints and engines must check them.
> It's
> > > > > > > previously
> > > > > > > > > > > > >>>>>> established
> > > > > > > > > > > > >>>>>>>> that
> > > > > > > > > > > > >>>>>>>>>> performance is just a software problem and
> > > > there is
> > > > > > no
> > > > > > > > > > > > >>>>> fundamental
> > > > > > > > > > > > >>>>>>>>>> performance advantage to the fixed size
> types
> > > in
> > > > > > > > > > > > >> Dremel-like
> > > > > > > > > > > > >>>>>> formats
> > > > > > > > > > > > >>>>>>>> like
> > > > > > > > > > > > >>>>>>>>>> Parquet.
> > > > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>> Best,
> > > > > > > > > > > > >>>>>>>>>> Will
> > > > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>> On Wed, 30 Sept 2026 at 16:08, Sebastian
> > > > Baunsgaard
> > > > > > <
> > > > > > > > > > > > >>>>>>>>> baunsgaard@apache.org
> > > > > > > > > > > > >>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>> wrote:
> > > > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>> After listening in on the last meeting
> > > > Wednesday, I
> > > > > > > > have
> > > > > > > > > > > > >>> been
> > > > > > > > > > > > >>>>>>>> thinking
> > > > > > > > > > > > >>>>>>>>>>> about this quite a bit, and thought I
> would
> > > > share
> > > > > > my
> > > > > > > 2
> > > > > > > > > > > > >>> cents.
> > > > > > > > > > > > >>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>> I suggest not requiring VECTOR elements
> to be
> > > > > > finite,
> > > > > > > > as
> > > > > > > > > > > > >>>>> proposed
> > > > > > > > > > > > >>>>>>> in
> > > > > > > > > > > > >>>>>>>>> the
> > > > > > > > > > > > >>>>>>>>>> PR
> > > > > > > > > > > > >>>>>>>>>>> <
> > > > https://github.com/apache/parquet-format/pull/624
> > > > > > >.
> > > > > > > > > > > > >>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>> While Divjot and others note the
> > > > specialization for
> > > > > > > > > > > > >> UTF-8,
> > > > > > > > > > > > >>>>>> Parquet
> > > > > > > > > > > > >>>>>>>>>> already
> > > > > > > > > > > > >>>>>>>>>>> defines DOUBLE and FLOAT as IEEE
> encodings
> > > > > > > > > > > > >>>>>>>>>>> <
> > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > >>>>
> > > > > > > > > > >
> > > > > > >
> > > https://github.com/apache/parquet-format/blob/master/README.md#types
> > > > > > > > > > > > >>>>>>>>>> .
> > > > > > > > > > > > >>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>> Making NaN or +/-Inf illegal inside
> VECTOR
> > > > would
> > > > > > be a
> > > > > > > > new
> > > > > > > > > > > > >>>>>>> restriction
> > > > > > > > > > > > >>>>>>>>> on
> > > > > > > > > > > > >>>>>>>>>>> those same underlying physical types.
> Scalar
> > > > > > > > FLOAT/DOUBLE
> > > > > > > > > > > > >>>>> columns
> > > > > > > > > > > > >>>>>>>> have
> > > > > > > > > > > > >>>>>>>>>>> never had such a rule.
> > > > > > > > > > > > >>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>> It would also not match how the systems
> that
> > > > > > actually
> > > > > > > > > > > > >>> produce
> > > > > > > > > > > > >>>>>> these
> > > > > > > > > > > > >>>>>>>>>> vectors
> > > > > > > > > > > > >>>>>>>>>>> treat floats. Most of them use IEEE
> floats,
> > > > e.g.
> > > > > > > Spark,
> > > > > > > > > > > > >>>> DuckDB,
> > > > > > > > > > > > >>>>>>>> NumPy,
> > > > > > > > > > > > >>>>>>>>>>> PyTorch, TensorFlow, or lower level CUDA,
> > > > oneMKL,
> > > > > > and
> > > > > > > > > > > > >>>> OpenBLAS.
> > > > > > > > > > > > >>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>> pgvector's "all elements must be finite"
> rule
> > > > is an
> > > > > > > > > > > > >> insert
> > > > > > > > > > > > >>>>>> policy.
> > > > > > > > > > > > >>>>>>>>> While
> > > > > > > > > > > > >>>>>>>>>> I
> > > > > > > > > > > > >>>>>>>>>>> think Parquet as a general-purpose format
> > > > should
> > > > > > > allow
> > > > > > > > > > > > >> them
> > > > > > > > > > > > >>>> out
> > > > > > > > > > > > >>>>>> of
> > > > > > > > > > > > >>>>>>>> the
> > > > > > > > > > > > >>>>>>>>>> box.
> > > > > > > > > > > > >>>>>>>>>>> It is a user system decision to disallow
> > > > writing
> > > > > > > these
> > > > > > > > > > > > >>> values
> > > > > > > > > > > > >>>>> or
> > > > > > > > > > > > >>>>>> to
> > > > > > > > > > > > >>>>>>>>> throw
> > > > > > > > > > > > >>>>>>>>>>> errors when encountering them.
> > > > > > > > > > > > >>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>> A type-level ban also does not keep
> Inf/NaN
> > > > out of
> > > > > > > > vector
> > > > > > > > > > > > >>>> math.
> > > > > > > > > > > > >>>>>>>> Finite
> > > > > > > > > > > > >>>>>>>>>>> coordinates still overflow. For instance:
> > > > > > magnitude =
> > > > > > > > > > > > >>>>>> ||[FLOAT_MAX,
> > > > > > > > > > > > >>>>>>>>>>> FLOAT_MAX]|| where every coordinate is
> finite
> > > > > > > overflows
> > > > > > > > > > > > >> to
> > > > > > > > > > > > >>>>> +Inf.
> > > > > > > > > > > > >>>>>>>>> PyTorch
> > > > > > > > > > > > >>>>>>>>>>> documents the same overflow:
> tensor([1e20,
> > > > > > > > 1e20]).norm()
> > > > > > > > > > > > >> is
> > > > > > > > > > > > >>>> inf
> > > > > > > > > > > > >>>>>> in
> > > > > > > > > > > > >>>>>>>>> fp32.
> > > > > > > > > > > > >>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>> A generated MAGNITUDE(vec) column would
> then
> > > > be Inf
> > > > > > > > while
> > > > > > > > > > > > >>> the
> > > > > > > > > > > > >>>>>>> VECTOR
> > > > > > > > > > > > >>>>>>>>>> itself
> > > > > > > > > > > > >>>>>>>>>>> was required to be finite. The ban only
> > > forbids
> > > > > > > storing
> > > > > > > > > > > > >>>>> specials
> > > > > > > > > > > > >>>>>> in
> > > > > > > > > > > > >>>>>>>> the
> > > > > > > > > > > > >>>>>>>>>>> coordinates, and does not prevent even
> simple
> > > > > > > > downstream
> > > > > > > > > > > > >>>>>>> computations
> > > > > > > > > > > > >>>>>>>>>> from
> > > > > > > > > > > > >>>>>>>>>>> producing them.
> > > > > > > > > > > > >>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>> I suggest keeping VECTOR as the type and
> add
> > > an
> > > > > > > > optional
> > > > > > > > > > > > >>> bool
> > > > > > > > > > > > >>>>>>>>>>> VectorType.finite (default false).
> Engines
> > > that
> > > > > > want
> > > > > > > > > > > > >>>>>> pgvector-like
> > > > > > > > > > > > >>>>>>>>>>> rejection can require the flag = true.
> > > Engines
> > > > that
> > > > > > > use
> > > > > > > > > > > > >>>> NumPy /
> > > > > > > > > > > > >>>>>>>>> PyTorch /
> > > > > > > > > > > > >>>>>>>>>>> Spark arrays can keep representable IEEE
> > > > values.
> > > > > > For
> > > > > > > > the
> > > > > > > > > > > > >>> same
> > > > > > > > > > > > >>>>>>>> reason, I
> > > > > > > > > > > > >>>>>>>>>> do
> > > > > > > > > > > > >>>>>>>>>>> not favor adding yet another type called
> > > > > > > FINITE_VECTOR.
> > > > > > > > > > > > >>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>> Best,
> > > > > > > > > > > > >>>>>>>>>>> Sebastian
> > > > > > > > > > > > >>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>> On Tue, Sep 29, 2026 at 2:03 PM Divjot
> Arora
> > > > via
> > > > > > dev
> > > > > > > <
> > > > > > > > > > > > >>>>>>>>>>> dev@parquet.apache.org>
> > > > > > > > > > > > >>>>>>>>>>> wrote:
> > > > > > > > > > > > >>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>> I agree with moving forward with the
> > > proposal
> > > > and
> > > > > > > > > > > > >>> renaming
> > > > > > > > > > > > >>>>>> VECTOR
> > > > > > > > > > > > >>>>>>>> to
> > > > > > > > > > > > >>>>>>>>>>>> FINITE_VECTOR or something
> > > > > > > > > > > > >>>>>>>>>>>> similar if others feel that better
> > > > communicates
> > > > > > its
> > > > > > > > > > > > >>>> intent. I
> > > > > > > > > > > > >>>>>> see
> > > > > > > > > > > > >>>>>>>>> value
> > > > > > > > > > > > >>>>>>>>>>> in
> > > > > > > > > > > > >>>>>>>>>>>> having a vector type that guarantees
> > > > > > > > > > > > >>>>>>>>>>>> its values are mathematically-correct
> > > vectors
> > > > for
> > > > > > a
> > > > > > > > few
> > > > > > > > > > > > >>>>>> reasons:
> > > > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>> 1. As others have mentioned, this is
> similar
> > > > to
> > > > > > the
> > > > > > > > > > > > >>> STRING
> > > > > > > > > > > > >>>>>>> logical
> > > > > > > > > > > > >>>>>>>>> type
> > > > > > > > > > > > >>>>>>>>>>>> mandating UTF-8.
> > > > > > > > > > > > >>>>>>>>>>>> 2. It is not possible to rely on
> statistics
> > > > for
> > > > > > > > > > > > >> detecting
> > > > > > > > > > > > >>>> NaN
> > > > > > > > > > > > >>>>>> or
> > > > > > > > > > > > >>>>>>>>> +-inf
> > > > > > > > > > > > >>>>>>>>>>>> values as all statistics are optional.
> > > > > > > > > > > > >>>>>>>>>>>> To my knowledge, there is no precedent
> for
> > > > > > requiring
> > > > > > > > > > > > >> that
> > > > > > > > > > > > >>>>>> writers
> > > > > > > > > > > > >>>>>>>>> emit
> > > > > > > > > > > > >>>>>>>>>>>> stats for specific logical types.
> > > > > > > > > > > > >>>>>>>>>>>> 3. Without statistics, NaN/inf values
> could
> > > > flow
> > > > > > > > > > > > >> through
> > > > > > > > > > > > >>>> the
> > > > > > > > > > > > >>>>>>> engine
> > > > > > > > > > > > >>>>>>>>> and
> > > > > > > > > > > > >>>>>>>>>>>> block them for specific functionality.
> > > > > > > > > > > > >>>>>>>>>>>> This seems to be Lance's behavior.
> However,
> > > I
> > > > > > don't
> > > > > > > > > > > > >> think
> > > > > > > > > > > > >>>>> this
> > > > > > > > > > > > >>>>>> is
> > > > > > > > > > > > >>>>>>>> as
> > > > > > > > > > > > >>>>>>>>>> easy
> > > > > > > > > > > > >>>>>>>>>>>> as it sounds and Lance has run
> > > > > > > > > > > > >>>>>>>>>>>> into several bugs due to this ([1] [2]
> for
> > > > > > > examples).
> > > > > > > > > > > > >>> Their
> > > > > > > > > > > > >>>>>>> Python
> > > > > > > > > > > > >>>>>>>>> [3]
> > > > > > > > > > > > >>>>>>>>>>> and
> > > > > > > > > > > > >>>>>>>>>>>> Rust [4] APIs default to erroring on
> > > > > > > > > > > > >>>>>>>>>>>> vectors that contain NaN values.
> > > > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>> I agree that Parquet is a
> general-purpose
> > > > format,
> > > > > > > but
> > > > > > > > I
> > > > > > > > > > > > >>>> don't
> > > > > > > > > > > > >>>>>>> think
> > > > > > > > > > > > >>>>>>>>>> this
> > > > > > > > > > > > >>>>>>>>>>>> implies the lack of domain-specific
> types.
> > > > > > > > > > > > >>>>>>>>>>>> Instead, this broadens the use cases
> where
> > > > Parquet
> > > > > > > > > > > > >> would
> > > > > > > > > > > > >>> be
> > > > > > > > > > > > >>>>>>>>> considered
> > > > > > > > > > > > >>>>>>>>>> a
> > > > > > > > > > > > >>>>>>>>>>>> good fit. I also fully agree with
> > > > > > > > > > > > >>>>>>>>>>>> continuing work on a general fixed-size
> list
> > > > type
> > > > > > if
> > > > > > > > we
> > > > > > > > > > > > >>>>> believe
> > > > > > > > > > > > >>>>>>>> there
> > > > > > > > > > > > >>>>>>>>>> are
> > > > > > > > > > > > >>>>>>>>>>>> non-vector use cases that would benefit
> > > > > > > > > > > > >>>>>>>>>>>> from such guarantees at the storage
> layer.
> > > > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>> [1]
> > > > > > https://github.com/lance-format/lance/pull/3648
> > > > > > > > > > > > >>>>>>>>>>>> [2]
> > > > > > > https://github.com/lance-format/lance/issues/8607
> > > > > > > > > > > > >>> (see
> > > > > > > > > > > > >>>>>>> point 2
> > > > > > > > > > > > >>>>>>>>> in
> > > > > > > > > > > > >>>>>>>>>>>> "root
> > > > > > > > > > > > >>>>>>>>>>>> cause")
> > > > > > > > > > > > >>>>>>>>>>>> [3]
> > > > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>
> > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > >>>>>>>
> > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > >>>>>
> > > > > > > > > > > > >>>>
> > > > > > > > > > > > >>>
> > > > > > > > > > > > >>
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > >
> > >
> https://github.com/lancedb/lancedb/blob/0327618a62483feb6dca2336549845e5887cd48f/python/python/lancedb/table.py#L4807-L4911
> > > > > > > > > > > > >>>>>>>>>>>> [4]
> > > > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>
> > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > >>>>>>>
> > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > >>>>>
> > > > > > > > > > > > >>>>
> > > > > > > > > > > > >>>
> > > > > > > > > > > > >>
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > >
> > >
> https://github.com/lancedb/lancedb/blob/0327618a62483feb6dca2336549845e5887cd48f/rust/lancedb/src/table/add_data.rs#L105-L110
> > > > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>> On Mon, Sep 28, 2026 at 6:19 PM Philipp
> > > > Fischbeck
> > > > > > > via
> > > > > > > > > > > > >>> dev <
> > > > > > > > > > > > >>>>>>>>>>>> dev@parquet.apache.org> wrote:
> > > > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>>> I also favor moving forward with the
> > > > proposal as
> > > > > > > is.
> > > > > > > > > > > > >> I
> > > > > > > > > > > > >>>> see
> > > > > > > > > > > > >>>>>>> strong
> > > > > > > > > > > > >>>>>>>>>> value
> > > > > > > > > > > > >>>>>>>>>>>> and
> > > > > > > > > > > > >>>>>>>>>>>>> use cases for the numeric vector type,
> and
> > > I
> > > > > > agree
> > > > > > > > > > > > >> with
> > > > > > > > > > > > >>>> the
> > > > > > > > > > > > >>>>>>>>>>> restrictions
> > > > > > > > > > > > >>>>>>>>>>>>> around NaN/Inf values and element
> types in
> > > > > > general.
> > > > > > > > > > > > >>>>> Precedent
> > > > > > > > > > > > >>>>>>>>> exists
> > > > > > > > > > > > >>>>>>>>>> in
> > > > > > > > > > > > >>>>>>>>>>>>> other types that restrict which
> physical
> > > type
> > > > > > > values
> > > > > > > > > > > > >>> are
> > > > > > > > > > > > >>>>>>>> logically
> > > > > > > > > > > > >>>>>>>>>>> valid
> > > > > > > > > > > > >>>>>>>>>>>>> (e.g., the geospatial types needing to
> be
> > > > > > > > > > > > >> WKB-encoded).
> > > > > > > > > > > > >>>> A
> > > > > > > > > > > > >>>>>>>> general
> > > > > > > > > > > > >>>>>>>>>>>>> fixed-size list also has value, but I
> think
> > > > it
> > > > > > > makes
> > > > > > > > > > > > >>>> sense
> > > > > > > > > > > > >>>>> to
> > > > > > > > > > > > >>>>>>>>>> separate
> > > > > > > > > > > > >>>>>>>>>>>> the
> > > > > > > > > > > > >>>>>>>>>>>>> two types. The semantics are relevant,
> and
> > > I
> > > > can
> > > > > > > see
> > > > > > > > > > > > >>> how
> > > > > > > > > > > > >>>>> the
> > > > > > > > > > > > >>>>>>> two
> > > > > > > > > > > > >>>>>>>>>> types
> > > > > > > > > > > > >>>>>>>>>>>>> might diverge further in the future
> > > regarding
> > > > > > > > > > > > >>> statistics
> > > > > > > > > > > > >>>> or
> > > > > > > > > > > > >>>>>>>>>> encodings.
> > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>>> I would be okay with renaming the
> vector
> > > > type,
> > > > > > but
> > > > > > > it
> > > > > > > > > > > > >>>> seems
> > > > > > > > > > > > >>>>>>> like
> > > > > > > > > > > > >>>>>>>>> the
> > > > > > > > > > > > >>>>>>>>>>>> VECTOR
> > > > > > > > > > > > >>>>>>>>>>>>> name is already used across other
> engines
> > > > > > mentioned
> > > > > > > > > > > > >> in
> > > > > > > > > > > > >>>> this
> > > > > > > > > > > > >>>>>>>> thread.
> > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>>> Best,
> > > > > > > > > > > > >>>>>>>>>>>>> Philipp
> > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>>> On Sat, Sep 26, 2026 at 4:37 PM Gang
> Wu <
> > > > > > > > > > > > >>>> ustcwg@gmail.com>
> > > > > > > > > > > > >>>>>>>> wrote:
> > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>>>> To separate orthogonal concerns, I
> feel we
> > > > > > should
> > > > > > > > > > > > >>> treat
> > > > > > > > > > > > >>>>> the
> > > > > > > > > > > > >>>>>>>>> general
> > > > > > > > > > > > >>>>>>>>>>>>>> FixedSizeList as the underlying
> physical
> > > > layout,
> > > > > > > > > > > > >> and
> > > > > > > > > > > > >>>>>> separate
> > > > > > > > > > > > >>>>>>>> it
> > > > > > > > > > > > >>>>>>>>>> from
> > > > > > > > > > > > >>>>>>>>>>>> the
> > > > > > > > > > > > >>>>>>>>>>>>>> discussion of a specialized VECTOR (or
> > > > broader
> > > > > > ML
> > > > > > > > > > > > >>>> tensor)
> > > > > > > > > > > > >>>>>>>> logical
> > > > > > > > > > > > >>>>>>>>>>> type.
> > > > > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>>>> Currently, the immediate need is to
> > > define a
> > > > > > > > > > > > >>>> specialized
> > > > > > > > > > > > >>>>>>> VECTOR
> > > > > > > > > > > > >>>>>>>>>> type
> > > > > > > > > > > > >>>>>>>>>>> so
> > > > > > > > > > > > >>>>>>>>>>>>>> that different engines and formats
> have a
> > > > common
> > > > > > > > > > > > >>>> semantic
> > > > > > > > > > > > >>>>>>>>> contract
> > > > > > > > > > > > >>>>>>>>>>> for
> > > > > > > > > > > > >>>>>>>>>>>>>> interoperability.
> > > > > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>>>> If the main point of contention is
> just
> > > the
> > > > > > > > > > > > >>>> NaN/Inf/null
> > > > > > > > > > > > >>>>>>>>>> limitations,
> > > > > > > > > > > > >>>>>>>>>>>>> maybe
> > > > > > > > > > > > >>>>>>>>>>>>>> we can resolve this by adding a
> > > > qualification to
> > > > > > > > > > > > >> the
> > > > > > > > > > > > >>>> name
> > > > > > > > > > > > >>>>>>> (like
> > > > > > > > > > > > >>>>>>>>>>> Andrew
> > > > > > > > > > > > >>>>>>>>>>>>>> suggested, e.g., VECTOR_EMBEDDING or
> > > > > > > FINITE_VECTOR)
> > > > > > > > > > > > >>> to
> > > > > > > > > > > > >>>>>> avoid
> > > > > > > > > > > > >>>>>>>>>>> confusion
> > > > > > > > > > > > >>>>>>>>>>>>> with
> > > > > > > > > > > > >>>>>>>>>>>>>> broader ML use cases. Alternatively,
> we
> > > > could
> > > > > > > > > > > > >>>> introduce a
> > > > > > > > > > > > >>>>>>>> boolean
> > > > > > > > > > > > >>>>>>>>>>> flag
> > > > > > > > > > > > >>>>>>>>>>>>>> (e.g., optional bool
> VectorType.finite) as
> > > > > > > proposed
> > > > > > > > > > > > >>>>>> earlier.
> > > > > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>>>> On Fri, Sep 25, 2026 at 3:52 AM Andrew
> > > Lamb
> > > > <
> > > > > > > > > > > > >>>>>>>>>> andrewlamb11@gmail.com>
> > > > > > > > > > > > >>>>>>>>>>>>>> wrote:
> > > > > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>>>>> I am a little confused about the
> points
> > > of
> > > > > > > > > > > > >>>> disagreement
> > > > > > > > > > > > >>>>>>> here.
> > > > > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>>>>> Is it that the name `VECTOR` is too
> > > > general for
> > > > > > > > > > > > >> the
> > > > > > > > > > > > >>>>>>> specific
> > > > > > > > > > > > >>>>>>>>> use
> > > > > > > > > > > > >>>>>>>>>>> case
> > > > > > > > > > > > >>>>>>>>>>>>> of
> > > > > > > > > > > > >>>>>>>>>>>>>>> fixed length arrays with non-finite
> > > values
> > > > used
> > > > > > > > > > > > >> for
> > > > > > > > > > > > >>>>>> vector
> > > > > > > > > > > > >>>>>>>>>>>> embeddings?
> > > > > > > > > > > > >>>>>>>>>>>>>> If
> > > > > > > > > > > > >>>>>>>>>>>>>>> so, maybe we could add a
> qualification to
> > > > the
> > > > > > > > > > > > >> name
> > > > > > > > > > > > >>>>>>>>>> (FINITE_VECTOR?
> > > > > > > > > > > > >>>>>>>>>>>>>>> FEATURE_VECTOR? VECTOR_EMBEDDING?)
> > > > > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>>>>> It seems like there is precedent for
> > > > defining
> > > > > > > > > > > > >>> logical
> > > > > > > > > > > > >>>>>> types
> > > > > > > > > > > > >>>>>>>>> that
> > > > > > > > > > > > >>>>>>>>>>>>> specify
> > > > > > > > > > > > >>>>>>>>>>>>>>> certain physical types and restrict
> their
> > > > > > > > > > > > >> contents
> > > > > > > > > > > > >>> to
> > > > > > > > > > > > >>>>>> help
> > > > > > > > > > > > >>>>>>>>>> readers
> > > > > > > > > > > > >>>>>>>>>>>>>>> interpret meaning, for example
> string[1]
> > > or
> > > > > > JSON;
> > > > > > > > > > > > >>>>>> Defining
> > > > > > > > > > > > >>>>>>>> one
> > > > > > > > > > > > >>>>>>>>>> for
> > > > > > > > > > > > >>>>>>>>>>>>>> vector
> > > > > > > > > > > > >>>>>>>>>>>>>>> embeddings seems reasonable enough
> to me.
> > > > > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>>>>> Andrew
> > > > > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>>>>> [1]:
> > > > > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>
> > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > >>>>>>>
> > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > >>>>>
> > > > > > > > > > > > >>>>
> > > > > > > > > > > > >>>
> > > > > > > > > > > > >>
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > >
> > >
> https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#string
> > > > > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>>>>> On Thu, Sep 24, 2026 at 3:34 PM
> Daniel
> > > > Weeks <
> > > > > > > > > > > > >>>>>>>>> dweeks@apache.org>
> > > > > > > > > > > > >>>>>>>>>>>>> wrote:
> > > > > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>>>>>>> Well, Parquet is not a
> single-purpose
> > > > format.
> > > > > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>>>>>>> Vector stores can choose whatever
> > > policy
> > > > fits
> > > > > > > > > > > > >>>> their
> > > > > > > > > > > > >>>>>>>> usage,
> > > > > > > > > > > > >>>>>>>>>> but
> > > > > > > > > > > > >>>>>>>>>>>>> we're
> > > > > > > > > > > > >>>>>>>>>>>>>>>>> talking about a general-purpose
> file
> > > > format
> > > > > > > > > > > > >>> that
> > > > > > > > > > > > >>>>> aims
> > > > > > > > > > > > >>>>>>> to
> > > > > > > > > > > > >>>>>>>> be
> > > > > > > > > > > > >>>>>>>>>>>> broadly
> > > > > > > > > > > > >>>>>>>>>>>>>>>>> applicable.
> > > > > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>>>>>> I agree that the format is intended
> to
> > > be
> > > > > > > > > > > > >> general
> > > > > > > > > > > > >>>>>>> purpose,
> > > > > > > > > > > > >>>>>>>>> but
> > > > > > > > > > > > >>>>>>>>>>>> types
> > > > > > > > > > > > >>>>>>>>>>>>>>> within
> > > > > > > > > > > > >>>>>>>>>>>>>>>> the format define specific
> > > representation
> > > > and
> > > > > > > > > > > > >>>>> semantics
> > > > > > > > > > > > >>>>>>> to
> > > > > > > > > > > > >>>>>>>>>>> achieve
> > > > > > > > > > > > >>>>>>>>>>>>> that
> > > > > > > > > > > > >>>>>>>>>>>>>>>> goal.
> > > > > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>>>>>>> "Vector" has a more general sense
> than
> > > > ML and
> > > > > > > > > > > > >>>> it's
> > > > > > > > > > > > >>>>>>> quite
> > > > > > > > > > > > >>>>>>>>>>> trivial
> > > > > > > > > > > > >>>>>>>>>>>> to
> > > > > > > > > > > > >>>>>>>>>>>>>>>>> produce vectors with non-finite
> values
> > > > using
> > > > > > > > > > > > >>> e.g.
> > > > > > > > > > > > >>>>>> NumPy
> > > > > > > > > > > > >>>>>>>> or
> > > > > > > > > > > > >>>>>>>>>>> Pandas
> > > > > > > > > > > > >>>>>>>>>>>>> (or
> > > > > > > > > > > > >>>>>>>>>>>>>>>>> any other piece of dataframe or
> > > > scientific
> > > > > > > > > > > > >>>>> computing
> > > > > > > > > > > > >>>>>>>>>> software).
> > > > > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>>>>>> The justification for restricting
> > > > non-finite
> > > > > > > > > > > > >>>> values
> > > > > > > > > > > > >>>>> is
> > > > > > > > > > > > >>>>>>>>>> directly
> > > > > > > > > > > > >>>>>>>>>>>>>> related
> > > > > > > > > > > > >>>>>>>>>>>>>>> to
> > > > > > > > > > > > >>>>>>>>>>>>>>>> the semantic meaning of a vector.
> > > Vectors
> > > > > > have
> > > > > > > > > > > > >>>>>> specific
> > > > > > > > > > > > >>>>>>>>>>>>>> characteristics
> > > > > > > > > > > > >>>>>>>>>>>>>>>> (dimension, magnitude, direction,
> etc).
> > > > > > > > > > > > >>> Non-finite
> > > > > > > > > > > > >>>>>>> values
> > > > > > > > > > > > >>>>>>>>> are
> > > > > > > > > > > > >>>>>>>>>>>>> invalid
> > > > > > > > > > > > >>>>>>>>>>>>>>> and
> > > > > > > > > > > > >>>>>>>>>>>>>>>> break those characteristics.  I'd
> liken
> > > > it to
> > > > > > > > > > > > >>>> storing
> > > > > > > > > > > > >>>>>>>>> non-utf8
> > > > > > > > > > > > >>>>>>>>>>>> values
> > > > > > > > > > > > >>>>>>>>>>>>>> in
> > > > > > > > > > > > >>>>>>>>>>>>>>> a
> > > > > > > > > > > > >>>>>>>>>>>>>>>> string and still calling it a
> string.
> > > > > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>>>>>> That also differentiates a vector
> from a
> > > > fixed
> > > > > > > > > > > > >>>> length
> > > > > > > > > > > > >>>>>>> list,
> > > > > > > > > > > > >>>>>>>>>> which
> > > > > > > > > > > > >>>>>>>>>>>> is
> > > > > > > > > > > > >>>>>>>>>>>>>> more
> > > > > > > > > > > > >>>>>>>>>>>>>>>> general and does not carry the same
> > > > semantic
> > > > > > > > > > > > >>>> meaning.
> > > > > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>>>>>> I also don't think we need to
> bundle all
> > > > > > > > > > > > >>> similarly
> > > > > > > > > > > > >>>>>> shaped
> > > > > > > > > > > > >>>>>>>> use
> > > > > > > > > > > > >>>>>>>>>>> cases
> > > > > > > > > > > > >>>>>>>>>>>>>> into
> > > > > > > > > > > > >>>>>>>>>>>>>>>> the same solution (e.g. flxed size
> > > list).
> > > > > > > > > > > > >> Vector
> > > > > > > > > > > > >>>> is
> > > > > > > > > > > > >>>>> a
> > > > > > > > > > > > >>>>>>> well
> > > > > > > > > > > > >>>>>>>>>>>>> established
> > > > > > > > > > > > >>>>>>>>>>>>>>>> data type at this point and aligns
> with
> > > > the
> > > > > > > > > > > > >>> general
> > > > > > > > > > > > >>>>>>> purpose
> > > > > > > > > > > > >>>>>>>>>>> nature
> > > > > > > > > > > > >>>>>>>>>>>> of
> > > > > > > > > > > > >>>>>>>>>>>>>> the
> > > > > > > > > > > > >>>>>>>>>>>>>>>> format (many other formats and
> databases
> > > > > > > > > > > > >> natively
> > > > > > > > > > > > >>>>>> support
> > > > > > > > > > > > >>>>>>>>>> this).
> > > > > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>>>>>> -Dan
> > > > > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>>>>>> On Thu, Sep 24, 2026 at 11:55 AM
> Antoine
> > > > > > > > > > > > >> Pitrou <
> > > > > > > > > > > > >>>>>>>>>>>> antoine@python.org>
> > > > > > > > > > > > >>>>>>>>>>>>>>>> wrote:
> > > > > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>>>>>>> Le 24/09/2026 à 20:11, Russell
> Spitzer
> > > a
> > > > > > > > > > > > >> écrit
> > > > > > > > > > > > >>> :
> > > > > > > > > > > > >>>>>>>>>>>>>>>>>> I think something I haven't seen
> yet
> > > is
> > > > the
> > > > > > > > > > > > >>>> clear
> > > > > > > > > > > > >>>>>>>>> use-case
> > > > > > > > > > > > >>>>>>>>>>> for
> > > > > > > > > > > > >>>>>>>>>>>>>>>> non-finite
> > > > > > > > > > > > >>>>>>>>>>>>>>>>>> values within a vector type.
> There has
> > > > > > > > > > > > >> been a
> > > > > > > > > > > > >>>> lot
> > > > > > > > > > > > >>>>>> of
> > > > > > > > > > > > >>>>>>>>>>>> theoretical
> > > > > > > > > > > > >>>>>>>>>>>>>>>>> discussion
> > > > > > > > > > > > >>>>>>>>>>>>>>>>>> here about use cases but what
> toolings
> > > > > > > > > > > > >>>> currently
> > > > > > > > > > > > >>>>>>>> produce
> > > > > > > > > > > > >>>>>>>>>>> these
> > > > > > > > > > > > >>>>>>>>>>>>>>> vectors
> > > > > > > > > > > > >>>>>>>>>>>>>>>>> and
> > > > > > > > > > > > >>>>>>>>>>>>>>>>>> how do they function in downstream
> > > > engines?
> > > > > > > > > > > > >>> I'd
> > > > > > > > > > > > >>>>>> like
> > > > > > > > > > > > >>>>>>> to
> > > > > > > > > > > > >>>>>>>>>> limit
> > > > > > > > > > > > >>>>>>>>>>>>>>>> introducing
> > > > > > > > > > > > >>>>>>>>>>>>>>>>>> types that have a wide range of
> values
> > > > > > > > > > > > >> which
> > > > > > > > > > > > >>>> are
> > > > > > > > > > > > >>>>>>> simply
> > > > > > > > > > > > >>>>>>>>> are
> > > > > > > > > > > > >>>>>>>>>>>>> failure
> > > > > > > > > > > > >>>>>>>>>>>>>>>> modes
> > > > > > > > > > > > >>>>>>>>>>>>>>>>>> of different flavors
> > > > > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>>>>>>> "Vector" has a more general sense
> than
> > > > ML and
> > > > > > > > > > > > >>>> it's
> > > > > > > > > > > > >>>>>>> quite
> > > > > > > > > > > > >>>>>>>>>>> trivial
> > > > > > > > > > > > >>>>>>>>>>>> to
> > > > > > > > > > > > >>>>>>>>>>>>>>>>> produce vectors with non-finite
> values
> > > > using
> > > > > > > > > > > > >>> e.g.
> > > > > > > > > > > > >>>>>> NumPy
> > > > > > > > > > > > >>>>>>>> or
> > > > > > > > > > > > >>>>>>>>>>> Pandas
> > > > > > > > > > > > >>>>>>>>>>>>> (or
> > > > > > > > > > > > >>>>>>>>>>>>>>>>> any other piece of dataframe or
> > > > scientific
> > > > > > > > > > > > >>>>> computing
> > > > > > > > > > > > >>>>>>>>>> software).
> > > > > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>>>>>>> It's not like the general concept
> of a
> > > > vector
> > > > > > > > > > > > >>> is
> > > > > > > > > > > > >>>> a
> > > > > > > > > > > > >>>>>> IETF
> > > > > > > > > > > > >>>>>>>>>>> standard
> > > > > > > > > > > > >>>>>>>>>>>> or
> > > > > > > > > > > > >>>>>>>>>>>>>>>>> anything, so analogies with JSON or
> > > UTF8
> > > > > > > > > > > > >> (where
> > > > > > > > > > > > >>>> the
> > > > > > > > > > > > >>>>>>>> values
> > > > > > > > > > > > >>>>>>>>>> are
> > > > > > > > > > > > >>>>>>>>>>>>>> expected
> > > > > > > > > > > > >>>>>>>>>>>>>>>>> to be valid according to the
> respective
> > > > > > > > > > > > >> specs)
> > > > > > > > > > > > >>> do
> > > > > > > > > > > > >>>>> not
> > > > > > > > > > > > >>>>>>>>> apply.
> > > > > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>>>>>>> This discussion would IMHO have a
> > > better
> > > > > > > > > > > > >> chance
> > > > > > > > > > > > >>>> of
> > > > > > > > > > > > >>>>>>>>> converging
> > > > > > > > > > > > >>>>>>>>>>> if
> > > > > > > > > > > > >>>>>>>>>>>> we
> > > > > > > > > > > > >>>>>>>>>>>>>>>>> separated the two orthogonal
> concerns
> > > of:
> > > > > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>>>>>>> 1. representing fixed-length
> > > list/vector
> > > > data
> > > > > > > > > > > > >>> in
> > > > > > > > > > > > >>>>>>> Parquet
> > > > > > > > > > > > >>>>>>>>>> (with
> > > > > > > > > > > > >>>>>>>>>>> a
> > > > > > > > > > > > >>>>>>>>>>>>>>>>> potential limitation to non-nested
> > > types)
> > > > > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>>>>>>> 2. representing the information
> that a
> > > > given
> > > > > > > > > > > > >>>> column
> > > > > > > > > > > > >>>>>> (or
> > > > > > > > > > > > >>>>>>>>>> column
> > > > > > > > > > > > >>>>>>>>>>>>> chunk,
> > > > > > > > > > > > >>>>>>>>>>>>>>>>> etc.) does not contain non-finite
> > > values
> > > > > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>>>>>>> ... both of which are useful on
> their
> > > own
> > > > > > > > > > > > >>> right,
> > > > > > > > > > > > >>>>> but
> > > > > > > > > > > > >>>>>> do
> > > > > > > > > > > > >>>>>>>> not
> > > > > > > > > > > > >>>>>>>>>>> have
> > > > > > > > > > > > >>>>>>>>>>>> to
> > > > > > > > > > > > >>>>>>>>>>>>>> be
> > > > > > > > > > > > >>>>>>>>>>>>>>>>> conflated and bundled together.
> > > > > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>>>>>>> Regards
> > > > > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>>>>>>> Antoine.
> > > > > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>
> > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > >>>>>>>
> > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > >>>>>
> > > > > > > > > > > > >>>>
> > > > > > > > > > > > >>>
> > > > > > > > > > > > >>
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] Numeric Vector type: physical representation (outcome of focused call)

[Fokko Driesprong <fo...@driesprong.frl>]

It feels like we're running in circles, let me add my 2c.

> I'd like to put the vector type on hold in Parquet until there's a general proposal for ML tensors and sparse, packed or quantised representations. That's where a new type would let engines do interoperable things they can't do today.

I'm afraid that's not how it works. There are folks that are willing to do the legwork for the VECTOR logical type. Sparse and Quantised representation sounds more like something where we want to focus on encoding, rather than a logical type.

> I'm asking for a workload that can't be done on LIST<FLOAT> now and would work with VECTOR.

We're all smart engineers, and we can make it work. But I think this is more about enforcing certain constraints on the data. Similar to the example of UTF8 and BYTES that was coined earlier. Or you can store UUIDs in a FIXED_LEN_BYTE_ARRAY, but there is a specific annotation for that as well. It is nice to have some semantical meaning to the types.

> If a finite fixed-length vector contract is useful to table formats, perhaps Iceberg is the place for it

I strongly feel that Parquet should be self-contained. As with the examples above, I feel this is something that is in line with Parquet.

Kind regards,
Fokko


Sent with Proton Mail secure email.

On Thursday, October 22nd, 2026 at 07:11, Will Edwards via dev <de...@parquet.apache.org> wrote:

> Hi Ryan,
> 
> Thanks for the detailed replies. Let me restate my position more directly,
> since I don't think I made it clear.
> 
> I'd like to put the vector type on hold in Parquet until there's a general
> proposal for ML tensors and sparse, packed or quantised representations.
> That's where a new type would let engines do interoperable things they
> can't do today.
> 
> My earlier question was what VECTOR makes possible that isn't possible
> today. The answers so far describe what the annotation asserts: that the
> data is a vector, that it has a fixed length, and that its values are
> finite and non-null. I'm asking for a workload that can't be done on
> LIST<FLOAT> now and would work with VECTOR. Jan asked the same question,
> and I don't think either of us has seen one yet.
> 
> For embeddings, LIST<FLOAT> is what is already widely used today, and it
> works. My compatibility concern is about writers. A tool that doesn't know
> VECTOR rewrites the file as plain LIST<FLOAT>. If engines then require the
> annotation, that data stops working. If they don't require it, the
> annotation adds little. A world where VECTOR exists and plain LIST<FLOAT>
> still works fully for ANN etc is a world that doesn't need VECTOR.
> 
> On optimisations, Dan framed this as semantics rather than performance, but
> you've listed decoding optimisations. In my experience readers can remove
> the cost of variable-length lists without any format change, and I'm happy
> to share the recipes with anyone interested. If fixed length is still the
> real gain, a fixed-size list gives it to everyone, NaN or not.
> 
> If table formats need a finite-vector contract, I think the table layer is
> a better home than the file format. An Iceberg type would apply equally to
> Parquet, Avro and ORC. I'd still be wary of Iceberg rejecting untagged
> columns, but it's a better place for that debate than Parquet.
> 
> Best,
> Will
> 
> On Thu, 22 Oct 2026 at 02:00, Ryan Blue <bl...@apache.org> wrote:
> 
> > Thanks, Curt. I wasn't aware of that behavior in MySQL. I was trying to
> > learn more and I don't see it in documentation either. Do you have any more
> > information about how MySQL handles this? One page makes it sound like it
> > may be platform dependent
> > <https://dev.mysql.com/doc/refman/26.7/en/problems-with-float.html>, where
> > some values produce "inf" and some produce "0".
> >
> > My original comment was referring to databases that allow users to specify
> > types that exclude non-finite values, which is what we would need in order
> > to support a finite float in table formats since we already support the
> > full range of values. If you know of any cases like that rather than just
> > limiting values, I'm curious. Thanks!
> >
> > Ryan
> >
> > On Wed, Oct 21, 2026 at 3:48 PM Curt Hagenlocher <cu...@hagenlocher.org>
> > wrote:
> >
> > > > I'm not aware of databases with finite floating point types.
> > >
> > > Microsoft SQL Server, MySQL and SAP ASE (née Sybase) do not allow
> > > storing non-finite floating-point values in their binary
> > > floating-point columns. They don't have a separate type which does
> > > allow this.
> > >
> > > On Wed, Oct 21, 2026 at 3:30 PM Ryan Blue <bl...@apache.org> wrote:
> > > >
> > > > Jan,
> > > >
> > > > I agree that we _could_ model vectors as an orthogonal finiteness
> > > property
> > > > for elements in a fixed-length list. I don't know how useful that would
> > > be.
> > > > I've not had anyone ask for that, and I'm not aware of databases with
> > > > finite floating point types. I think it makes more sense to move
> > forward
> > > > with the existing FINITE_VECTOR that is proposed.
> > > >
> > > > For the logical type argument, it sounds like the issue is that you
> > want
> > > to
> > > > fall back to readers treating vectors as LIST. I don't see a good way
> > to
> > > > make that happen, but it also isn't needed by the table format use case
> > > > because we are introducing this as a new type. Do you have a use case
> > > where
> > > > you are going to mix FINITE_VECTOR with LIST files and this will cause
> > a
> > > > problem?
> > > >
> > > > I don't think that we've had an expectation for logical types in the
> > past
> > > > that they produce a particular in-memory representation. Maybe we could
> > > > allow setting both `LIST` and `FINITE_VECTOR` logical types at the same
> > > > time? This kind of compatibility hasn't been a priority as far as I'm
> > > > aware, but if it is important to you we can consider it.
> > > >
> > > > Ryan
> > > >
> > > > On Wed, Oct 21, 2026 at 1:13 PM Jan Finis <jp...@gmail.com> wrote:
> > > >
> > > > >  assuming what will be proposed is a new logical type FINITE_VECTOR
> > > that
> > > > > annotates the same column that would have been annotated as LIST
> > > before:
> > > > >
> > > > > What I find somewhat suboptimal of FINITE_VECTOR is that finiteness
> > is
> > > part
> > > > > of the vector type. Isn't the finiteness rather a property of the
> > > contained
> > > > > floats? Such a modeling would be more composable, as it would for
> > > example
> > > > > also allow to have normal (non-vector) float columns that are
> > > restricted to
> > > > > be finite. So, if we do think that we want finiteness in the type
> > > system, I
> > > > > would rather cut it into two parts: A fixed size vector concept, and
> > a
> > > new
> > > > > way of labeling float columns finite. The vector type would then just
> > > boil
> > > > > down to be a fixed size list / we could just give the list type an
> > > optional
> > > > > fixed size field. Then, the column contained within it would be a
> > float
> > > > > tagged as finite.
> > > > >
> > > > > It was mentioned that a logical type annotating a list of float would
> > > be
> > > > > forward compatible, but - depending on the modeling - that is not
> > > > > necessarily true: as logical types in Parquet do not support
> > > subtyping, a
> > > > > column cannot be a list of float and a (finite) vector at the same
> > > time,
> > > > > but only one of the two. It depends on how the type will be modeled
> > in
> > > > > thrift in the end, but if it is a new union member of `LogicalType`,
> > > then
> > > > > legacy engines will not understand that such a vector can also just
> > be
> > > > > interpreted as a list of floats. They will just see a nested column
> > > type of
> > > > > a logical type they don't understand, so they will likely reject
> > > reading
> > > > > this column (at least that's what the engine I'm maintaining would be
> > > > > doing).
> > > > >
> > > > > In contrast, if we just give the list type an optional fixed size and
> > > float
> > > > > an optional finiteness, then the resulting thrift is actually fully
> > > forward
> > > > > compatible: Legacy engines will just ignore the fixed length and
> > treat
> > > the
> > > > > list as variable length and they will ignore the finiteness and treat
> > > the
> > > > > element type as float.
> > > > >
> > > > > (A side note, which most people here may already be aware of, but
> > it's
> > > good
> > > > > to repeat it: The not-NULL property doesn't need new type support at
> > > all.
> > > > > The normal type system of list already allows setting the contained
> > > element
> > > > > type to be optional or required. If it is set to be required, then
> > > NULLs
> > > > > are not allowed.)
> > > > >
> > > > > Cheers,
> > > > > Jan
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > Am Di., 20. Okt. 2026 um 20:37 Uhr schrieb Russell Spitzer <
> > > > > russell.spitzer@gmail.com>:
> > > > >
> > > > > > I support proceeding with the two types. I honestly think the cost
> > of
> > > > > > adding new logical types is overestimated. If we have as many
> > > interested
> > > > > > parties as have appeared in this thread we already know there is a
> > > large
> > > > > > group of engineers ready to do the work and engines ready to
> > consume
> > > (and
> > > > > > produce) the type. The fact that implementations currently use
> > > > > heuristics,
> > > > > > sampling (I loved Gunnar's talk on his hardwood work on this at CoC
> > > > > > Glasgow) or custom annotations to achieve the same performance
> > > benefits
> > > > > > makes me think we should just create the type and move on.
> > > > > >
> > > > > > On Mon, Oct 19, 2026 at 7:21 PM Ryan Blue <bl...@apache.org> wrote:
> > > > > >
> > > > > > > An annotation on list<float> is useful because:
> > > > > > > - We want to correctly annotate what is being stored
> > > > > > > - We want to unlock optimizations when each non-null list has the
> > > same
> > > > > > > number of elements
> > > > > > > - We want to have writer guarantees
> > > > > > >
> > > > > > > I don't think that logical type compatibility is a good argument.
> > > We
> > > > > want
> > > > > > > to avoid a case where we have to treat all binary like a string.
> > > And
> > > > > > > because we are introducing this in table formats after
> > > standardizing it
> > > > > > > here, we can do that reliably.
> > > > > > >
> > > > > > > As I said in my last email, I think there's a difference between
> > > > > > finiteness
> > > > > > > and general "invariants" you might want. Vectors aren't vectors
> > if
> > > they
> > > > > > > don't have a length and a direction. Other invariants don't work
> > > the
> > > > > same
> > > > > > > way. A zero-length vector is still a valid vector and can be
> > used,
> > > just
> > > > > > not
> > > > > > > for certain distance metrics.
> > > > > > >
> > > > > > > I also want to clarify that when we looked into this more closely
> > > and
> > > > > > > recognized the mismatch between the two separate use cases, I
> > > offered
> > > > > to
> > > > > > > split out the finite vector work so that we didn't cause any
> > > headaches
> > > > > > for
> > > > > > > people who are interested in fixed-length list. It sounds like
> > not
> > > > > > > splitting out the use cases is causing a lot of headache. Vectors
> > > for
> > > > > our
> > > > > > > use case need a finiteness guarantee at write time, and
> > > fixed-length
> > > > > > lists
> > > > > > > need to store NaN and other values.
> > > > > > >
> > > > > > > It's okay to have two types and we should for two different use
> > > cases.
> > > > > > > Otherwise, we are forcing one to make an unnecessary compromise.
> > I
> > > > > don't
> > > > > > > want to force everyone to exclude NaN and infinite values from
> > > > > > fixed-length
> > > > > > > lists. Seems reasonable that we should also be able to have
> > vectors
> > > > > that
> > > > > > > exclude those values, since that is how they are used and we have
> > > to
> > > > > also
> > > > > > > guarantee it for exchanging data with other existing databases.
> > > > > > >
> > > > > > > Ryan
> > > > > > >
> > > > > > > On Mon, Oct 19, 2026 at 3:12 AM Will Edwards via dev <
> > > > > > > dev@parquet.apache.org>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > It reads well. Three small fixes:
> > > > > > > >
> > > > > > > > 1. **"back in May or so"** is ambiguous. The
> > > > > > fixed-size-list-for-tensors
> > > > > > > > framing is from Rok's opening post in May 2024. The May 2026
> > > design
> > > > > doc
> > > > > > > was
> > > > > > > > already mostly about performance and layout. "Originally"
> > avoids
> > > > > anyone
> > > > > > > > correcting the date.
> > > > > > > > 2. **"Spark's sparse feature vectors routinely use NaN"** isn't
> > > quite
> > > > > > > > right. The NaN-for-missing behaviour comes from
> > > `VectorAssembler` and
> > > > > > > > `Imputer` and applies to Spark ML vectors generally, dense or
> > > sparse.
> > > > > > > > "Spark's feature vectors, dense and sparse," fixes it.
> > > > > > > > 3. There are typos: "Iirc" should be "IIRC", and "Spark,
> > > BigQuery and
> > > > > > > > DuckDB and pretty much…" has a doubled "and".
> > > > > > > >
> > > > > > > > The last sentence of the invariants paragraph could be clearer.
> > > > > > "Although
> > > > > > > > that doesn't mean readers shouldn't check on read" has a double
> > > > > > negative
> > > > > > > > and is easy to misread. "Though careful readers will still
> > check
> > > on
> > > > > > read"
> > > > > > > > is simpler.
> > > > > > > >
> > > > > > > > With those applied:
> > > > > > > >
> > > > > > > > ---
> > > > > > > >
> > > > > > > > Hi all,
> > > > > > > >
> > > > > > > > The scope of this discussion keeps moving :). IIRC it
> > originally
> > > was
> > > > > > > about
> > > > > > > > fixed-size lists for ML tensors, became a performance question,
> > > and
> > > > > is
> > > > > > > now
> > > > > > > > about whether vectors are mathematically valid. Each step made
> > > sense,
> > > > > > > but I
> > > > > > > > don't think we have consensus on where we've ended up?
> > > > > > > >
> > > > > > > > I think there is much useful work in what we left behind. Many
> > ML
> > > > > > vectors
> > > > > > > > aren't embeddings for search. E.g. Spark's feature vectors,
> > > dense and
> > > > > > > > sparse, routinely use NaN to mark missing values, and quantised
> > > > > formats
> > > > > > > > pack codes with scales in ways Parquet can't describe today.
> > > > > > ML-oriented
> > > > > > > > formats like Lance are moving towards tensors, quantisation and
> > > > > > indexes.
> > > > > > > A
> > > > > > > > type that told engines how to interpret that kind of data would
> > > > > enable
> > > > > > > new
> > > > > > > > things. An annotation on LIST<FLOAT> doesn't.
> > > > > > > >
> > > > > > > > My big concern with the current focus on embeddings needing a
> > > logical
> > > > > > > type
> > > > > > > > is compatibility. Today LIST<FLOAT> is the common format for
> > > > > > embeddings,
> > > > > > > > and Spark, BigQuery, DuckDB and pretty much every other library
> > > I can
> > > > > > > think
> > > > > > > > of offer distance functions on plain lists, so any engine can
> > > use any
> > > > > > > file.
> > > > > > > > If engines start requiring a VECTOR annotation, existing files
> > > and
> > > > > > > anything
> > > > > > > > rewritten by a tool that doesn't know the type stop working. If
> > > they
> > > > > > > don't
> > > > > > > > require it, the annotation adds nothing. We've seen this before
> > > with
> > > > > > > > strings written without the UTF8 annotation, which engines
> > still
> > > have
> > > > > > to
> > > > > > > > cope with years later.
> > > > > > > >
> > > > > > > > On invariants, I agree some users may want the writer to check
> > > what
> > > > > > they
> > > > > > > > write as they write it, but that can be a writer option rather
> > > than a
> > > > > > > type.
> > > > > > > > Readers that want evidence could get it from extra statistics
> > or
> > > > > > > metadata,
> > > > > > > > though careful readers will still check on read.
> > > > > > > >
> > > > > > > > If a finite fixed-length vector contract is useful to table
> > > formats,
> > > > > > > > perhaps Iceberg is the place for it, with Parquet storing plain
> > > > > lists.
> > > > > > > I'd
> > > > > > > > raise the same worry there, though: if tables start rejecting
> > > > > untagged
> > > > > > > > columns, we've just moved the split up a layer.
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Will
> > > > > > > >
> > > > > > > > On Sat, 17 Oct 2026 at 00:10, Ryan Blue <bl...@apache.org>
> > wrote:
> > > > > > > >
> > > > > > > > > I agree with much of what Dan said, but I'll add some to this
> > > > > thread
> > > > > > as
> > > > > > > > > well.
> > > > > > > > >
> > > > > > > > > First, I want to address the arguments that were pointed out:
> > > > > > > > > 1. Other vector databases have finiteness requirements
> > > > > > > > > 2. "Vectors are the hot stuff right now" -- let's jump on the
> > > > > > bandwagon
> > > > > > > > > 3. A general solution will take too long, finiteness will
> > make
> > > > > faster
> > > > > > > > > progress
> > > > > > > > > 4. To give a guarantee to readers that no INF/NaN/NULL will
> > be
> > > in
> > > > > the
> > > > > > > > > vector
> > > > > > > > >
> > > > > > > > > I don't think these represent our actual motivation or
> > > position,
> > > > > > which
> > > > > > > > > makes them function as a straw-man. This is probably not what
> > > was
> > > > > > > > > intended, but I want to make sure that the misunderstanding
> > > > > > > > > doesn't continue. For one thing, we aren't saying that we
> > need
> > > > > finite
> > > > > > > > > vectors because they are "hot stuff".
> > > > > > > > >
> > > > > > > > > We want to build tools that use them on top of Parquet. I
> > > think it
> > > > > > > should
> > > > > > > > > also be clear that we aren't advocating for finite elements
> > > blindly
> > > > > > > just
> > > > > > > > > because other engines do (cargo cult design!). There are real
> > > > > > > > > interoperability issues if we choose to go a different
> > > direction so
> > > > > > > this
> > > > > > > > is
> > > > > > > > > a "Chesterton's Fence" situation. It's important to
> > understand
> > > why
> > > > > > > > > those databases
> > > > > > > > > made that choice. I've also pointed this out as confirmation
> > > that
> > > > > > we're
> > > > > > > > > going down a reasonable and established path.
> > > > > > > > >
> > > > > > > > > The third argument sounds like a simple misunderstanding.
> > We're
> > > > > > > pursuing
> > > > > > > > a
> > > > > > > > > logical type right now because that's forward compatible and
> > we
> > > > > want
> > > > > > to
> > > > > > > > > label the data properly and set expectations for the type.
> > > Physical
> > > > > > > type
> > > > > > > > > changes break forward compatibility and need to be released
> > in
> > > a
> > > > > new
> > > > > > > > format
> > > > > > > > > version . I think this is the source of the confusion since
> > it
> > > has
> > > > > > been
> > > > > > > > in
> > > > > > > > > the same discussion, but the choice about finiteness is
> > > separate.
> > > > > > > > >
> > > > > > > > > That brings us to the last argument, whether there is value
> > in
> > > > > > writers
> > > > > > > > > participating in guarantees. I think there's value in adding
> > > > > > > requirements
> > > > > > > > > for writers. It's up to readers how they want to use those
> > > > > > > requirements,
> > > > > > > > > but a writer requirement allows a policy of "garbage in
> > garbage
> > > > > out"
> > > > > > > > rather
> > > > > > > > > than being forced to handle bad data. Why would a reader need
> > > to
> > > > > run
> > > > > > > > > validation on every element just in case a writer had a bug?
> > > And
> > > > > why
> > > > > > > > should
> > > > > > > > > the potential for writer bugs mean that it has to be the
> > > reader's
> > > > > > > > problem?
> > > > > > > > > If a writer produces bad data, we can absolutely reject it
> > and
> > > tell
> > > > > > the
> > > > > > > > > writer to fix the bug.
> > > > > > > > >
> > > > > > > > > I'm also puzzled by the assertion that writer guarantees have
> > > > > value,
> > > > > > > but
> > > > > > > > > only if there are other, more specific guarantees in
> > addition.
> > > This
> > > > > > > seems
> > > > > > > > > to cede the point that a contract between readers and writers
> > > makes
> > > > > > > > sense,
> > > > > > > > > but only if it is a _specific_ contract. But the proposed
> > > additions
> > > > > > > don't
> > > > > > > > > make sense in the same way.
> > > > > > > > >
> > > > > > > > > Take, for example, the suggestion to also disallow
> > zero-length
> > > > > > vectors.
> > > > > > > > At
> > > > > > > > > a glance this seems similar, but I think it assumes a
> > specific
> > > > > > context
> > > > > > > > and
> > > > > > > > > is better suited to be a constraint for some use cases rather
> > > than
> > > > > > part
> > > > > > > > of
> > > > > > > > > what is considered valid for the type. That's because
> > > zero-length
> > > > > > > vectors
> > > > > > > > > are valid vectors with a length. You can calculate Euclidean
> > or
> > > > > > > Manhattan
> > > > > > > > > distance and use them in many contexts. They don't work if
> > > you're
> > > > > > > > > normalizing to a length of 1, but it's a choice to normalize
> > > and
> > > > > use
> > > > > > a
> > > > > > > > dot
> > > > > > > > > product to calculate cosine distance. And at this point,
> > we're
> > > > > > talking
> > > > > > > > > about the usefulness of restrictions for the type, not a
> > > Platonic
> > > > > > > ideal.
> > > > > > > > >
> > > > > > > > > I think the restriction that makes the most sense is that
> > > vectors
> > > > > > have
> > > > > > > a
> > > > > > > > > length and can be compared in distance calculations. 0-length
> > > > > vectors
> > > > > > > are
> > > > > > > > > valid because they point somewhere and have a length. We
> > > shouldn't
> > > > > > > > restrict
> > > > > > > > > uses to only cosine distance over normalized vectors. Those
> > > are use
> > > > > > > case
> > > > > > > > > dependent choices and there are other valid ones. I also
> > don't
> > > > > think
> > > > > > > that
> > > > > > > > > the model used to produce a vector is part of the vector
> > type,
> > > just
> > > > > > > like
> > > > > > > > > the method used to produce a metric doesn't define its type:
> > > cosine
> > > > > > > > > similarity produces values between -1 and 1, but we store it
> > > in a
> > > > > > > > `float`,
> > > > > > > > > not a `float(cosine)`.
> > > > > > > > >
> > > > > > > > > Why does it make sense to have a finiteness restriction as
> > > part of
> > > > > > the
> > > > > > > > > type? Because that's what a (mathematical) vector is. A
> > simple
> > > > > search
> > > > > > > > gives
> > > > > > > > > a pretty common definition: "A vector is an object that has
> > > both
> > > > > > > > magnitude
> > > > > > > > > and a direction". If a dimension/element is NaN, null, or
> > > infinite,
> > > > > > the
> > > > > > > > > vector doesn't point anywhere and it doesn't have a
> > magnitude.
> > > Not
> > > > > > only
> > > > > > > > can
> > > > > > > > > it not be used, it doesn't meet the most basic definition of
> > > the
> > > > > > type.
> > > > > > > > >
> > > > > > > > > Ryan
> > > > > > > > >
> > > > > > > > > On Thu, Oct 15, 2026 at 1:07 AM Jan Finis <jpfinis@gmail.com
> > >
> > > > > wrote:
> > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > Daniel,
> > > > > > > > > >
> > > > > > > > > > alright, so no physical optimizations intended.  We only
> > want
> > > > > this
> > > > > > > for
> > > > > > > > > > semantic reasons, not for any significant physical
> > > optimization.
> > > > > If
> > > > > > > > we're
> > > > > > > > > > all on the same page here, that is good 👍.
> > > > > > > > > >
> > > > > > > > > > To phrase my remaining problem with this more concretely:
> > > > > > > > > > *Let's say we add this type. What cool new stuff can we now
> > > do
> > > > > that
> > > > > > > we
> > > > > > > > > > couldn't do before?*
> > > > > > > > > > That is the litmus test that a type has to pass to be
> > useful
> > > > > IMHO.
> > > > > > > And
> > > > > > > > > here
> > > > > > > > > > I can't come up with a tangible use case that wasn't
> > possible
> > > > > > before.
> > > > > > > > > > Especially if we one day have fixed size lists in Parquet,
> > > the
> > > > > > > > remaining
> > > > > > > > > > niche for this type becomes so small, that the added
> > > complexity
> > > > > > isn't
> > > > > > > > > worth
> > > > > > > > > > the benefit IMHO.
> > > > > > > > > >
> > > > > > > > > > As I guess most people on here are active maintainers of a
> > > > > Parquet
> > > > > > > > > > implementation: Concretely, what would you be using this
> > > type for
> > > > > > if
> > > > > > > we
> > > > > > > > > had
> > > > > > > > > > it? Would you allow math only on fields tagged with this
> > > type?
> > > > > What
> > > > > > > if
> > > > > > > > > that
> > > > > > > > > > math yields INF or NaN? Not allowed?
> > > > > > > > > >
> > > > > > > > > > Especially if we also discuss having both VECTOR and
> > > > > FINITE_VECTOR.
> > > > > > > How
> > > > > > > > > > will you guys wire this up with the type systems of your
> > > engines?
> > > > > > Do
> > > > > > > > your
> > > > > > > > > > engines also have two types here? Do you type-decay a
> > > > > FINITE_VECTOR
> > > > > > > to
> > > > > > > > a
> > > > > > > > > > VECTOR once someone does a multiplication that could yield
> > > INF?
> > > > > By
> > > > > > > > > itself,
> > > > > > > > > > a FINITE_VECTOR sounds good. Holistically looked at from a
> > > type
> > > > > > > system
> > > > > > > > > > perspective, it comes with a lot of complexity. Do the
> > > benefits
> > > > > > > really
> > > > > > > > > > outweigh the complexity here? That's why I would rather
> > have
> > > the
> > > > > > bare
> > > > > > > > > > minimum: A fixed size array (list) of floats or at least
> > > just a
> > > > > > > vector
> > > > > > > > > > without finite. Then all these detailed questions go away.
> > > > > > > > > >
> > > > > > > > > > I'll take this one step further... Why is there any debate
> > > about
> > > > > > this
> > > > > > > > at
> > > > > > > > > > > all? Why can't a type simply be added to the
> > > *specification* as
> > > > > > an
> > > > > > > > > > > administrative detail and be done?
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Andrew,
> > > > > > > > > >
> > > > > > > > > > Parquet is supposed to be a lingua franca of data. The
> > > > > > specification
> > > > > > > is
> > > > > > > > > the
> > > > > > > > > > authoritative truth for this language and all
> > > implementations are
> > > > > > > > > expected
> > > > > > > > > > to implement as much as possible from this spec, so
> > anything
> > > in
> > > > > the
> > > > > > > > spec
> > > > > > > > > > does causes implementation effort. So just adding anything
> > to
> > > > > this
> > > > > > > spec
> > > > > > > > > > without debate is not a good idea. It is even the other way
> > > > > around:
> > > > > > > We
> > > > > > > > > need
> > > > > > > > > > to have way higher rigor with the spec than we need to have
> > > with
> > > > > > any
> > > > > > > > > > implementations. Implementations can be changed, the spec
> > > needs
> > > > > to
> > > > > > be
> > > > > > > > > > stable.
> > > > > > > > > >
> > > > > > > > > > A tag-only type that doesn't come with any restrictions can
> > > be
> > > > > > added
> > > > > > > > with
> > > > > > > > > > little effort. But here we want to tie restrictions to it
> > (no
> > > > > > > NaN/Inf),
> > > > > > > > > so
> > > > > > > > > > we would expect implementations to enforce these. So there
> > is
> > > > > some
> > > > > > > > > > implementation effort expected here.
> > > > > > > > > >
> > > > > > > > > > Also, even if we only wanted a tag type, I would still
> > argue
> > > that
> > > > > > you
> > > > > > > > > > shouldn't add those without a debate. What if A wants XML,
> > B
> > > > > wants
> > > > > > > > YAML,
> > > > > > > > > C
> > > > > > > > > > wants quaternions, D wants address, E wants phone number.
> > If
> > > we
> > > > > > just
> > > > > > > > > added
> > > > > > > > > > any types that are "just tags", Parquet logical types would
> > > just
> > > > > > > > become a
> > > > > > > > > > trash dump of any type that anyone ever thought to be
> > > useful. So
> > > > > > > > rigorous
> > > > > > > > > > assessment of each new type candidate is necessary.
> > > > > > > > > >
> > > > > > > > > > Finally, many engines couple parquet with another type
> > system
> > > > > > (e.g.,
> > > > > > > > > Arrow
> > > > > > > > > > or SQL). So even with a tag only type, you at least still
> > > have
> > > > > the
> > > > > > > > > > implementation burden of wiring this up with your engine's
> > > type
> > > > > > > system.
> > > > > > > > > >
> > > > > > > > > > So, just because logical types are "just" logical, that
> > > doesn't
> > > > > > mean
> > > > > > > we
> > > > > > > > > can
> > > > > > > > > > add them without any rigor.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > A few naive questions:
> > > > > > > > > > > 1. Would a physical definition with NANs & Inf+/- allow
> > > > > efficient
> > > > > > > > > > > encodings like ALP?
> > > > > > > > > > > 2. If the physical representation allows values that make
> > > > > > encodings
> > > > > > > > > like
> > > > > > > > > > > ALP impossible, can a tighter Logical type allow
> > different
> > > > > > > encodings?
> > > > > > > > > > > 3. Can we RLE or binpack a vector?
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > The simple general answer here is that all these things do
> > > not
> > > > > > > require
> > > > > > > > a
> > > > > > > > > > logical type. Writers can already do any such optimization
> > > if the
> > > > > > > > actual
> > > > > > > > > > data contains no NaNs & Infs. All these physical
> > > optimizations
> > > > > just
> > > > > > > > > require
> > > > > > > > > > no actual NaNs/Infs to be present in a page, they don't
> > need
> > > a
> > > > > > > > guarantee
> > > > > > > > > > that the whole file won't contain any.
> > > > > > > > > >
> > > > > > > > > > Cheers,
> > > > > > > > > > Jan
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Am Do., 15. Okt. 2026 um 04:28 Uhr schrieb <
> > > > > > kurtiscwright@gmail.com
> > > > > > > >:
> > > > > > > > > >
> > > > > > > > > > > Hi Everyone,
> > > > > > > > > > >
> > > > > > > > > > > Glad to see a Vector type being added. I have a few use
> > > cases I
> > > > > > > want
> > > > > > > > to
> > > > > > > > > > > pursue once we have an agreed upon definition.
> > > > > > > > > > >
> > > > > > > > > > > In regards to that definition discussion. I feel
> > > comfortable
> > > > > with
> > > > > > > the
> > > > > > > > > > idea
> > > > > > > > > > > of the semantic meaning Dan raises though I can’t shake a
> > > > > slight
> > > > > > > > > > discomfort
> > > > > > > > > > > (maybe FOMO) with limiting values in the physical
> > > definition.
> > > > > > > > > > >
> > > > > > > > > > > A few naive questions:
> > > > > > > > > > > 1. Would a physical definition with NANs & Inf+/- allow
> > > > > efficient
> > > > > > > > > > > encodings like ALP?
> > > > > > > > > > > 2. If the physical representation allows values that make
> > > > > > encodings
> > > > > > > > > like
> > > > > > > > > > > ALP impossible, can a tighter Logical type allow
> > different
> > > > > > > encodings?
> > > > > > > > > > > 3. Can we RLE or binpack a vector?
> > > > > > > > > > >
> > > > > > > > > > > Best Regards,
> > > > > > > > > > > Kurtis C. Wright
> > > > > > > > > > > > On Oct 14, 2026, at 12:45, Daniel Weeks <
> > > dweeks@apache.org>
> > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > Hey Jan,
> > > > > > > > > > > >
> > > > > > > > > > > > There's a lot that I disagree with respect to the
> > > framing of
> > > > > > > these
> > > > > > > > > > > > arguments and how we should be thinking about
> > introducing
> > > > > > vector
> > > > > > > > as a
> > > > > > > > > > > > logical type.  There's a repeated tendency to let the
> > > > > physical
> > > > > > > > > > > > representation or performance drive discussion around
> > > types
> > > > > as
> > > > > > > > > opposed
> > > > > > > > > > to
> > > > > > > > > > > > what it represents.
> > > > > > > > > > > >
> > > > > > > > > > > > In parquet, establishing semantic meaning is the
> > purpose
> > > of
> > > > > > > logical
> > > > > > > > > > > types.
> > > > > > > > > > > > The logical type separates how to interpret a value
> > from
> > > its
> > > > > > > > physical
> > > > > > > > > > > > representation.  We can represent a timestamp with
> > INT64
> > > or
> > > > > > even
> > > > > > > a
> > > > > > > > > > string
> > > > > > > > > > > > value, so why have a type for it? Even though a
> > > > > > FIXED_SIZED_LIST
> > > > > > > > and
> > > > > > > > > > > VECTOR
> > > > > > > > > > > > may have the exact same representation and performance,
> > > they
> > > > > > have
> > > > > > > > > > > different
> > > > > > > > > > > > semantic meaning.  Databases and engines across the
> > > ecosystem
> > > > > > > > already
> > > > > > > > > > > > recognize this differentiation and have separate
> > > > > > representations.
> > > > > > > > > The
> > > > > > > > > > > > choice to separate these isn't because the discussion
> > is
> > > too
> > > > > > > > tedious;
> > > > > > > > > > > > they're fundamentally different things.
> > > > > > > > > > > >
> > > > > > > > > > > > The finite requirement is what makes a vector a
> > > reflection of
> > > > > > > what
> > > > > > > > it
> > > > > > > > > > > > represents.  As we've discussed in the thread,
> > non-finite
> > > > > > values
> > > > > > > in
> > > > > > > > > the
> > > > > > > > > > > > representation violate the properties of a vector and
> > the
> > > > > value
> > > > > > > > > ceases
> > > > > > > > > > to
> > > > > > > > > > > > be a vector.  On the other hand, a 0 vector is still a
> > > vector
> > > > > > > (you
> > > > > > > > > can
> > > > > > > > > > > > argue the utility of it, but it is a valid vector).
> > > > > > > > Implementations
> > > > > > > > > > can
> > > > > > > > > > > > choose what they validate or how to handle execution
> > > errors;
> > > > > > > that's
> > > > > > > > > an
> > > > > > > > > > > > implementation concern, not a requirement.  Operations
> > > > > > performed
> > > > > > > > on a
> > > > > > > > > > > > vector depend on the context in which they are used,
> > but
> > > the
> > > > > > > > > > > representation
> > > > > > > > > > > > doesn't need to solve scenarios where that application
> > > > > doesn't
> > > > > > > make
> > > > > > > > > > > sense.
> > > > > > > > > > > >
> > > > > > > > > > > > The guarantees or context needed to use a vector do not
> > > need
> > > > > to
> > > > > > > be
> > > > > > > > > > > inherent
> > > > > > > > > > > > in the type for it to be a useful type.  We don't need
> > > the
> > > > > > > currency
> > > > > > > > > > used
> > > > > > > > > > > > for decimal or whether a floating point is a normalized
> > > value
> > > > > > as
> > > > > > > > part
> > > > > > > > > > of
> > > > > > > > > > > > the type to make use of those types.  This conflates
> > the
> > > use
> > > > > > > cases
> > > > > > > > > with
> > > > > > > > > > > the
> > > > > > > > > > > > type system and unnecessarily attempts to draw the
> > > context
> > > > > into
> > > > > > > the
> > > > > > > > > > type
> > > > > > > > > > > > definition.  Similarly, you don't need to capture
> > > > > > normalization,
> > > > > > > > > model,
> > > > > > > > > > > > etc. as part of the type for it to be useful.
> > > > > > > > > > > >
> > > > > > > > > > > > So the reason we want to add a vector type (or
> > > FINITE_VECTOR)
> > > > > > is
> > > > > > > > > > because
> > > > > > > > > > > we
> > > > > > > > > > > > want to store vectors with the semantic meaning and
> > > > > > requirements
> > > > > > > > > based
> > > > > > > > > > on
> > > > > > > > > > > > what they represent.
> > > > > > > > > > > >
> > > > > > > > > > > > -Dan
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >> On Wed, Oct 14, 2026 at 1:22 AM Jan Finis <
> > > > > jpfinis@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > > > > >>
> > > > > > > > > > > >> A bit late to the party, but here we go:
> > > > > > > > > > > >>
> > > > > > > > > > > >> I have the strong sentiment that the FINITE_VECTOR
> > type
> > > as
> > > > > it
> > > > > > is
> > > > > > > > > > > currently
> > > > > > > > > > > >> being discussed will not be very useful and just
> > > increase
> > > > > > > > > > implementation
> > > > > > > > > > > >> complexity. The arguments for the type are all quite
> > > weak
> > > > > > > (details
> > > > > > > > > see
> > > > > > > > > > > >> below). The FINITE part doesn't seem to yield any
> > value
> > > and
> > > > > is
> > > > > > > > not a
> > > > > > > > > > > useful
> > > > > > > > > > > >> restriction by itself. The vector (i.e. fixed length)
> > > part
> > > > > > > should
> > > > > > > > > > > rather be
> > > > > > > > > > > >> generalized instead of making a one-off thing. In
> > > general,
> > > > > the
> > > > > > > > > > existing
> > > > > > > > > > > >> types are already working well for uncompressed
> > > vectors. The
> > > > > > > real
> > > > > > > > > > > challenge
> > > > > > > > > > > >> that would bring real benefits is supporting
> > > quantization.
> > > > > > This
> > > > > > > is
> > > > > > > > > > where
> > > > > > > > > > > >> you get orders of magnitude improvements to storage
> > > space
> > > > > and
> > > > > > > > > > processing
> > > > > > > > > > > >> speed.
> > > > > > > > > > > >>
> > > > > > > > > > > >> Let's first discuss the FINITE part. So far, the only
> > > > > > arguments
> > > > > > > I
> > > > > > > > > have
> > > > > > > > > > > read
> > > > > > > > > > > >> for this are:
> > > > > > > > > > > >>
> > > > > > > > > > > >> 1. many other vector databases have it
> > > > > > > > > > > >> 2. vectors are the hot stuff right now, so Parquet
> > > better
> > > > > > boast
> > > > > > > > > having
> > > > > > > > > > > >> "native" type support for vectors
> > > > > > > > > > > >> 3. by focussing on this common special case, we can
> > make
> > > > > rapid
> > > > > > > > > > progress,
> > > > > > > > > > > >> while a general solution would take too long to
> > specify
> > > > > > > > > > > >> 4. to give a guarantee to readers that no INF/NaN/NULL
> > > will
> > > > > be
> > > > > > > in
> > > > > > > > > the
> > > > > > > > > > > >> vector
> > > > > > > > > > > >>
> > > > > > > > > > > >> the first two arguments are quite weak by themselves.
> > > Vector
> > > > > > > > > databases
> > > > > > > > > > > are
> > > > > > > > > > > >> very specialized engines. Such engines can of course
> > be
> > > very
> > > > > > > > > > > opinionated on
> > > > > > > > > > > >> their data types. A general format like Parquet must
> > > instead
> > > > > > > weigh
> > > > > > > > > > > whether
> > > > > > > > > > > >> a special type is worth the added complexity over a
> > more
> > > > > > > general,
> > > > > > > > > > widely
> > > > > > > > > > > >> applicable type. In this case, it is IMHO not at all.
> > > And
> > > > > just
> > > > > > > > > having
> > > > > > > > > > a
> > > > > > > > > > > >> type called "vector" for advertisement reasons to make
> > > > > Parquet
> > > > > > > > > appear
> > > > > > > > > > > more
> > > > > > > > > > > >> modern seems a non-goal in the first place.
> > > > > > > > > > > >>
> > > > > > > > > > > >> The third "time to market" argument is valid for
> > systems
> > > > > that
> > > > > > > > don't
> > > > > > > > > > > need a
> > > > > > > > > > > >> stable interface. It is a very dangerous attitude for
> > > > > formats
> > > > > > > like
> > > > > > > > > > > Parquet
> > > > > > > > > > > >> that must be stable long term and should have a
> > > > > consolidated,
> > > > > > > > > concise,
> > > > > > > > > > > >> non-redundant type system. For such long term storage
> > > > > formats,
> > > > > > > we
> > > > > > > > > need
> > > > > > > > > > > to
> > > > > > > > > > > >> take the time to build the right thing, as any
> > > shortcuts we
> > > > > > take
> > > > > > > > > will
> > > > > > > > > > > stay
> > > > > > > > > > > >> as blemishes in the format forever. If I read the
> > > discussion
> > > > > > > > > > correctly,
> > > > > > > > > > > >> this is actually the main argument as of now: We have
> > > found
> > > > > > that
> > > > > > > > it
> > > > > > > > > is
> > > > > > > > > > > hard
> > > > > > > > > > > >> to align on how a fixed size list type should look
> > > like, so
> > > > > we
> > > > > > > > want
> > > > > > > > > to
> > > > > > > > > > > go
> > > > > > > > > > > >> the easier route of a specialized type. This seems
> > like
> > > a
> > > > > > > > > > > >> dangerous shortcut fueled by feature recency bias. Ask
> > > > > > yourself:
> > > > > > > > > Would
> > > > > > > > > > > we
> > > > > > > > > > > >> want this, if Parquet already had a fixed size list
> > > type?
> > > > > > > > > > > >>
> > > > > > > > > > > >> So finally, the only technical argument is the fourth
> > > one:
> > > > > "It
> > > > > > > > > signals
> > > > > > > > > > > >> semantics and ensures that no bad data lands in the
> > > > > format". I
> > > > > > > see
> > > > > > > > > two
> > > > > > > > > > > big
> > > > > > > > > > > >> problems with this:
> > > > > > > > > > > >>
> > > > > > > > > > > >> 1. Readers *still* need to test for NaN / INF because
> > > > > Parquet
> > > > > > > > files
> > > > > > > > > > may
> > > > > > > > > > > be
> > > > > > > > > > > >> corrupt. Nothing can stop a malicious or buggy writer
> > > from
> > > > > > > > > physically
> > > > > > > > > > > >> writing NaN or INF. *So you save no single line of
> > code
> > > in a
> > > > > > > > reader.
> > > > > > > > > > You
> > > > > > > > > > > >> save no branches, as you still need to handle the
> > > INF/NaN
> > > > > > case.*
> > > > > > > > So
> > > > > > > > > > this
> > > > > > > > > > > >> doesn't help with performance at all. The only
> > > difference
> > > > > is,
> > > > > > > that
> > > > > > > > > the
> > > > > > > > > > > >> error message changes from "found an invalid INF value
> > > in
> > > > > the
> > > > > > > > data"
> > > > > > > > > to
> > > > > > > > > > > >> "Your parquet file seems to be corrupt". It is
> > actually
> > > > > quite
> > > > > > > the
> > > > > > > > > > > opposite,
> > > > > > > > > > > >> readers and writers get more complex having to handle
> > > this
> > > > > new
> > > > > > > > type.
> > > > > > > > > > > >>
> > > > > > > > > > > >> If you now argue that it's not about performance, it's
> > > about
> > > > > > > > defined
> > > > > > > > > > > >> semantics and erroring out early in writers, then the
> > > second
> > > > > > > > problem
> > > > > > > > > > > >> arises:
> > > > > > > > > > > >>
> > > > > > > > > > > >> 2. The proposed no-NaN/INF/NULL guarantee is just one
> > of
> > > > > > > multiple
> > > > > > > > > > > >> guarantees that you need for sensible vector
> > > processing. If
> > > > > > "no
> > > > > > > > > > > >> INF/NaN/NULL" was the only guarantee you needed, I
> > would
> > > > > agree
> > > > > > > > that
> > > > > > > > > > > >> FINITE_VECTOR is useful, but it is by no means the
> > only
> > > > > > > guarantee
> > > > > > > > > you
> > > > > > > > > > > need,
> > > > > > > > > > > >> instead, depending on the distance metric, you also
> > > need the
> > > > > > > > > following
> > > > > > > > > > > >> guarantees. E.g.:
> > > > > > > > > > > >>
> > > > > > > > > > > >>   - No vector may be the 0 vector, otherwise, e.g.,
> > > cosine
> > > > > > > > > similarity
> > > > > > > > > > is
> > > > > > > > > > > >>   undefined. Cosine similarity is often used as
> > > default, so
> > > > > > this
> > > > > > > > is
> > > > > > > > > > > >> actually
> > > > > > > > > > > >>   quite important in practice.
> > > > > > > > > > > >>   - If we want to use a dot product, then all vectors
> > > need
> > > > > to
> > > > > > > > > > guarantee
> > > > > > > > > > > a
> > > > > > > > > > > >>   length of 1. This would actually be quite useful for
> > > > > > > performance
> > > > > > > > > > > >> reasons.
> > > > > > > > > > > >>   - The vectors must come from the same model as the
> > > vector
> > > > > > used
> > > > > > > > for
> > > > > > > > > > the
> > > > > > > > > > > >>   search. Otherwise, the search yields a numeric
> > > result, but
> > > > > > > that
> > > > > > > > > > result
> > > > > > > > > > > >> has
> > > > > > > > > > > >>   no sensible meaning.
> > > > > > > > > > > >>
> > > > > > > > > > > >> So, what use is a type that just guarantees part of
> > the
> > > > > things
> > > > > > > you
> > > > > > > > > > need
> > > > > > > > > > > for
> > > > > > > > > > > >> vector search but not others? You still need
> > additional
> > > > > > > guarantees
> > > > > > > > > in
> > > > > > > > > > > >> addition to that type. What makes INF/NaN so important
> > > in
> > > > > > > > comparison
> > > > > > > > > > to,
> > > > > > > > > > > >> say, "no 0 vector present", that it warrants an own
> > > type,
> > > > > > while
> > > > > > > > the
> > > > > > > > > > > other
> > > > > > > > > > > >> guarantees do not? If we really want this type as a
> > > basis
> > > > > for
> > > > > > > > > > > cross-engine
> > > > > > > > > > > >> sharing of vectors, then it needs to give *all*
> > > guarantees.
> > > > > > > > > > > >>
> > > > > > > > > > > >> I'm not against a specialized vector type if it really
> > > > > carries
> > > > > > > > *all*
> > > > > > > > > > > >> information to be useful by itself. If it carried all
> > > that
> > > > > > > > > > information,
> > > > > > > > > > > >> then a reading engine could indeed validate that only
> > > > > > meaningful
> > > > > > > > > > vector
> > > > > > > > > > > >> searches are executed. But as long as it is just a
> > > > > half-baked
> > > > > > > > thing,
> > > > > > > > > > it
> > > > > > > > > > > is
> > > > > > > > > > > >> not useful by itself and just a burden on each
> > > implementer
> > > > > and
> > > > > > > > > > > maintainer.
> > > > > > > > > > > >> If we need extra information anyway, why not just
> > stick
> > > to a
> > > > > > > > general
> > > > > > > > > > > type
> > > > > > > > > > > >> like VECTOR (without FINITE).
> > > > > > > > > > > >>
> > > > > > > > > > > >> If you're a proponent of FINITE_VECTOR, ask yourself
> > (or
> > > > > tell
> > > > > > > us)
> > > > > > > > > what
> > > > > > > > > > > >> advantage you will *really* get over just a
> > (potentially
> > > > > fixed
> > > > > > > > size)
> > > > > > > > > > > List
> > > > > > > > > > > >> of Float? As discussed, it's not performance. It's
> > also
> > > not
> > > > > a
> > > > > > > > > semantic
> > > > > > > > > > > >> marker that is sufficient by itself. It's just a small
> > > hint
> > > > > > that
> > > > > > > > > > allows
> > > > > > > > > > > >> writers to error out once they see a NaN or INF in the
> > > > > input,
> > > > > > > > while
> > > > > > > > > > they
> > > > > > > > > > > >> will still happily write a bogus 0 vector. This seems
> > > to be
> > > > > of
> > > > > > > > zero
> > > > > > > > > > > >> benefit.
> > > > > > > > > > > >>
> > > > > > > > > > > >> Anecdotally, we currently handle petabytes of vectors
> > in
> > > > > > > > production
> > > > > > > > > in
> > > > > > > > > > > >> Data360. We simply store them in a List of Float, with
> > > an
> > > > > > > > optimized
> > > > > > > > > > > R-Level
> > > > > > > > > > > >> reader implementation that rapidly checks that all
> > > vectors
> > > > > are
> > > > > > > the
> > > > > > > > > > same
> > > > > > > > > > > >> length. This solution works just fine. The overhead of
> > > > > R-level
> > > > > > > > > > decoding
> > > > > > > > > > > is
> > > > > > > > > > > >> barely measurable. The check is super fast and
> > R-levels
> > > are
> > > > > > tiny
> > > > > > > > in
> > > > > > > > > > > >> comparison to the payload vectors, so also the added
> > > I/O is
> > > > > > > > > > negligible.
> > > > > > > > > > > >> This is also why I'm so firm in my sentiment: We are
> > > > > running a
> > > > > > > > > > > production
> > > > > > > > > > > >> grade system processing trillions of vectors right now
> > > and
> > > > > see
> > > > > > > no
> > > > > > > > > big
> > > > > > > > > > > >> issues with the types that Parquet currently provides.
> > > > > > > > > > > >>
> > > > > > > > > > > >> As I don't just want to be a naysayer and point out
> > > problems
> > > > > > > with
> > > > > > > > > the
> > > > > > > > > > > >> current design, here are the directions that I would
> > > propose
> > > > > > > > indeed:
> > > > > > > > > > > >>
> > > > > > > > > > > >>   - a general fixed size list type. Don't do shortcuts
> > > just
> > > > > > > > because
> > > > > > > > > > the
> > > > > > > > > > > >>   discussion is too tedious.
> > > > > > > > > > > >>   - a vector type that indeed carries enough
> > > information to
> > > > > be
> > > > > > > > > really
> > > > > > > > > > > >>   useful for vector search by itself. (e.g., model
> > name
> > > /
> > > > > > > version,
> > > > > > > > > > > >> optional
> > > > > > > > > > > >>   guarantees for no 0 vector / normalized vector
> > > length).
> > > > > With
> > > > > > > > such
> > > > > > > > > a
> > > > > > > > > > > >> type,
> > > > > > > > > > > >>   you could really do well defined vector searches
> > even
> > > if
> > > > > the
> > > > > > > > > writing
> > > > > > > > > > > >> engine
> > > > > > > > > > > >>   is completely different to the reading one.
> > > > > > > > > > > >>   - Standardize quantizations. This will be a real
> > game
> > > > > > changer
> > > > > > > > for
> > > > > > > > > > > >>   performance.
> > > > > > > > > > > >>
> > > > > > > > > > > >> Cheers,
> > > > > > > > > > > >> Jan
> > > > > > > > > > > >>
> > > > > > > > > > > >>
> > > > > > > > > > > >>
> > > > > > > > > > > >>
> > > > > > > > > > > >>
> > > > > > > > > > > >>
> > > > > > > > > > > >>> Am Mi., 14. Okt. 2026 um 03:36 Uhr schrieb Gang Wu <
> > > > > > > > > ustcwg@gmail.com
> > > > > > > > > > >:
> > > > > > > > > > > >>>
> > > > > > > > > > > >>> I think separating them into FINITE_VECTOR and
> > > > > > > FIXED_LENGTH_LIST
> > > > > > > > > is a
> > > > > > > > > > > >> good
> > > > > > > > > > > >>> idea, though I think renaming FINITE_VECTOR to VECTOR
> > > looks
> > > > > > > more
> > > > > > > > > > > natural.
> > > > > > > > > > > >>> Should we move forward with proposing
> > > FINITE_VECTOR/VECTOR
> > > > > > > first?
> > > > > > > > > > > >>>
> > > > > > > > > > > >>> On Tue, Oct 13, 2026 at 6:26 PM Alkis Evlogimenos via
> > > dev <
> > > > > > > > > > > >>> dev@parquet.apache.org> wrote:
> > > > > > > > > > > >>>
> > > > > > > > > > > >>>> I suggest we go with VECTOR for the finite vector to
> > > keep
> > > > > > the
> > > > > > > > name
> > > > > > > > > > the
> > > > > > > > > > > >>> same
> > > > > > > > > > > >>>> as other engines/type systems. If we want a
> > > non-finite one
> > > > > > in
> > > > > > > > the
> > > > > > > > > > > >> future
> > > > > > > > > > > >>> we
> > > > > > > > > > > >>>> can name that something else or we add fixed size
> > list
> > > > > that
> > > > > > > will
> > > > > > > > > > serve
> > > > > > > > > > > >>> the
> > > > > > > > > > > >>>> same purpose without a special name.
> > > > > > > > > > > >>>>
> > > > > > > > > > > >>>>
> > > > > > > > > > > >>>> On Thu, Oct 8, 2026 at 10:20 PM Ryan Blue <
> > > > > blue@apache.org>
> > > > > > > > > wrote:
> > > > > > > > > > > >>>>
> > > > > > > > > > > >>>>> A logical type would:
> > > > > > > > > > > >>>>> - Describe data for users and readers to understand
> > > what
> > > > > > the
> > > > > > > > file
> > > > > > > > > > > >>>> contains
> > > > > > > > > > > >>>>> - Signal that repetition and definition levels will
> > > > > > reflect a
> > > > > > > > > fixed
> > > > > > > > > > > >>>> number
> > > > > > > > > > > >>>>> of elements, which implementations can use to
> > > optimize
> > > > > > > decoding
> > > > > > > > > > > >>>>> - Guarantee that there are no null elements
> > (because
> > > the
> > > > > > type
> > > > > > > > > > cannot
> > > > > > > > > > > >> be
> > > > > > > > > > > >>>>> applied to a list with optional elements)
> > > > > > > > > > > >>>>> - [FINITE_VECTOR] Guarantee that no elements are
> > NaN
> > > or
> > > > > > > > Infinity
> > > > > > > > > > > >> values
> > > > > > > > > > > >>>> so
> > > > > > > > > > > >>>>> that the vectors have a finite length and are
> > > suitable
> > > > > for
> > > > > > > > > distance
> > > > > > > > > > > >>>>> calculations
> > > > > > > > > > > >>>>>
> > > > > > > > > > > >>>>> The finite value restriction is a restriction on
> > > writers
> > > > > to
> > > > > > > > > prevent
> > > > > > > > > > > >> bad
> > > > > > > > > > > >>>>> data from being written.
> > > > > > > > > > > >>>>>
> > > > > > > > > > > >>>>> Ignoring logical types that are not understood is a
> > > rule
> > > > > > for
> > > > > > > > > > > >> readers. A
> > > > > > > > > > > >>>>> reader can handle vectors and pass it on to another
> > > > > layer.
> > > > > > > For
> > > > > > > > > > > >>> instance,
> > > > > > > > > > > >>>> a
> > > > > > > > > > > >>>>> Parquet reader that doesn't support geometry passes
> > > > > values
> > > > > > as
> > > > > > > > > > > >> `binary`.
> > > > > > > > > > > >>>>> That binary value can be passed to a geometry UDF,
> > > even
> > > > > if
> > > > > > > the
> > > > > > > > > > > >>> underlying
> > > > > > > > > > > >>>>> engine and Parquet reader doesn't have support.
> > > > > > > > > > > >>>>>
> > > > > > > > > > > >>>>> A tool that doesn't understand FINITE_VECTOR can
> > > > > similarly
> > > > > > > > pass a
> > > > > > > > > > > >> list
> > > > > > > > > > > >>> of
> > > > > > > > > > > >>>>> floats along to a UDF. But it is still important to
> > > the
> > > > > > user
> > > > > > > > that
> > > > > > > > > > bad
> > > > > > > > > > > >>>>> values are not present in the data. In your
> > example,
> > > bad
> > > > > > > values
> > > > > > > > > are
> > > > > > > > > > > >>>> vectors
> > > > > > > > > > > >>>>> that cause the centroid calculation to fail, like
> > > vectors
> > > > > > > with
> > > > > > > > > > > >> infinite
> > > > > > > > > > > >>>>> length.
> > > > > > > > > > > >>>>>
> > > > > > > > > > > >>>>> Ryan
> > > > > > > > > > > >>>>>
> > > > > > > > > > > >>>>> On Thu, Oct 8, 2026 at 12:54 PM Will Edwards via
> > dev
> > > <
> > > > > > > > > > > >>>>> dev@parquet.apache.org>
> > > > > > > > > > > >>>>> wrote:
> > > > > > > > > > > >>>>>
> > > > > > > > > > > >>>>>> Could we clarify the use cases for a VECTOR
> > > annotation
> > > > > on
> > > > > > > > > > > >>> LIST<FLOAT>?
> > > > > > > > > > > >>>>>> Specifically, what does it make possible that's
> > > > > impossible
> > > > > > > > > today,
> > > > > > > > > > > >> and
> > > > > > > > > > > >>>>> what
> > > > > > > > > > > >>>>>> does it make easier that's currently hard?
> > > > > > > > > > > >>>>>>
> > > > > > > > > > > >>>>>> I'm struggling to find a strong answer. I'd like
> > > every
> > > > > > > engine
> > > > > > > > to
> > > > > > > > > > > >>> offer
> > > > > > > > > > > >>>>>> vector functions on lists of floats without
> > needing
> > > a
> > > > > new
> > > > > > > > > logical
> > > > > > > > > > > >>> type.
> > > > > > > > > > > >>>>>>
> > > > > > > > > > > >>>>>> To be clear, I'm still enthusiastic about a vector
> > > type
> > > > > > that
> > > > > > > > > > > >>> describes
> > > > > > > > > > > >>>>> how
> > > > > > > > > > > >>>>>> to unpack and scale composite representations
> > > (quantised
> > > > > > > codes
> > > > > > > > > > plus
> > > > > > > > > > > >>>>> scales,
> > > > > > > > > > > >>>>>> codebooks, etc.), in the direction ML is going.
> > That
> > > > > adds
> > > > > > > > > > > >> information
> > > > > > > > > > > >>>>>> engines don't have. A label on LIST<FLOAT>
> > doesn't:
> > > > > > engines
> > > > > > > > > > already
> > > > > > > > > > > >>>>>> understand the physical type, and many already
> > offer
> > > > > > vector
> > > > > > > > > > > >> functions
> > > > > > > > > > > >>>> on
> > > > > > > > > > > >>>>>> it. I don't see a performance benefit, and
> > > NaN/null/Inf
> > > > > > > checks
> > > > > > > > > are
> > > > > > > > > > > >>>> cheap
> > > > > > > > > > > >>>>>> enough that engines should do them anyway. The
> > label
> > > > > also
> > > > > > > > > doesn't
> > > > > > > > > > > >> say
> > > > > > > > > > > >>>>> what
> > > > > > > > > > > >>>>>> you can do with the column. Knowing that a column
> > > holds
> > > > > > > > > embeddings
> > > > > > > > > > > >>>>> doesn't
> > > > > > > > > > > >>>>>> say how they can be used, just as zipcode and age
> > > are
> > > > > both
> > > > > > > > > > integers
> > > > > > > > > > > >>> but
> > > > > > > > > > > >>>>>> multiplying them is meaningless.
> > > > > > > > > > > >>>>>>
> > > > > > > > > > > >>>>>> My concern is the opposite effect. We're
> > > establishing
> > > > > that
> > > > > > > > > engines
> > > > > > > > > > > >>>> should
> > > > > > > > > > > >>>>>> ignore logical types they don't understand. So a
> > > tool
> > > > > that
> > > > > > > > > doesn't
> > > > > > > > > > > >>> know
> > > > > > > > > > > >>>>>> VECTOR will read the column as LIST<FLOAT>, let
> > the
> > > user
> > > > > > > > compute
> > > > > > > > > > > >>>>> centroids
> > > > > > > > > > > >>>>>> (say), and write plain LIST<FLOAT> back out. If
> > > other
> > > > > > > engines
> > > > > > > > > > > >>> restrict
> > > > > > > > > > > >>>>>> embedding functions to VECTOR-annotated columns,
> > > that
> > > > > file
> > > > > > > > stops
> > > > > > > > > > > >>>> working
> > > > > > > > > > > >>>>>> with them, through no fault of the user. An
> > > annotation
> > > > > > that
> > > > > > > > > > doesn't
> > > > > > > > > > > >>>>> change
> > > > > > > > > > > >>>>>> what the data is shouldn't be able to make it
> > > unusable.
> > > > > > > > > > > >>>>>>
> > > > > > > > > > > >>>>>> Am I missing a use case?
> > > > > > > > > > > >>>>>>
> > > > > > > > > > > >>>>>> Best, Will
> > > > > > > > > > > >>>>>>
> > > > > > > > > > > >>>>>> On Thu, 8 Oct 2026 at 19:53, Steven Wu <
> > > > > > > stevenz3wu@gmail.com>
> > > > > > > > > > > >> wrote:
> > > > > > > > > > > >>>>>>
> > > > > > > > > > > >>>>>>> I also think a FINITE_VECTOR type brings a lot of
> > > > > values
> > > > > > to
> > > > > > > > > many
> > > > > > > > > > > >>> use
> > > > > > > > > > > >>>>>> cases.
> > > > > > > > > > > >>>>>>> It is would be great to make progress on the
> > > > > > FINITE_VECTOR,
> > > > > > > > and
> > > > > > > > > > > >> not
> > > > > > > > > > > >>>> to
> > > > > > > > > > > >>>>>> fit
> > > > > > > > > > > >>>>>>> every use cases into the FIXED_LENGTH_LIST.
> > > > > > > > > > > >>>>>>>
> > > > > > > > > > > >>>>>>> On Mon, Oct 5, 2026 at 1:26 PM Ryan Blue <
> > > > > > blue@apache.org>
> > > > > > > > > > > >> wrote:
> > > > > > > > > > > >>>>>>>
> > > > > > > > > > > >>>>>>>> Going through this thread again, it looks like
> > > there
> > > > > > are a
> > > > > > > > few
> > > > > > > > > > > >>>> things
> > > > > > > > > > > >>>>>>> that
> > > > > > > > > > > >>>>>>>> people generally agree on:
> > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > >>>>>>>> - In the short term, we want to introduce a
> > > logical
> > > > > type
> > > > > > > or
> > > > > > > > > > > >>> types;
> > > > > > > > > > > >>>> a
> > > > > > > > > > > >>>>>>> better
> > > > > > > > > > > >>>>>>>> physical representation can be done later. We
> > can
> > > > > focus
> > > > > > on
> > > > > > > > > > > >>> logical
> > > > > > > > > > > >>>>>> types.
> > > > > > > > > > > >>>>>>>> - There is precedent in logical types for
> > > restricting
> > > > > > > > values,
> > > > > > > > > > > >>> like
> > > > > > > > > > > >>>>> UTF8
> > > > > > > > > > > >>>>>>>> strings to contain only UTF-8 codepoints. I
> > think
> > > we
> > > > > > also
> > > > > > > > > agree
> > > > > > > > > > > >>>> that
> > > > > > > > > > > >>>>> we
> > > > > > > > > > > >>>>>>>> could restrict vector elements to non-nested or
> > > > > > primitive
> > > > > > > > > > > >> types.
> > > > > > > > > > > >>>>>>>> - There are many examples of databases that
> > > restrict
> > > > > > their
> > > > > > > > > > > >> vector
> > > > > > > > > > > >>>>> types
> > > > > > > > > > > >>>>>>> to
> > > > > > > > > > > >>>>>>>> non-null, finite elements.
> > > > > > > > > > > >>>>>>>> - Some people want to have write-time guarantees
> > > that
> > > > > > > > vectors
> > > > > > > > > > > >>> have
> > > > > > > > > > > >>>>> only
> > > > > > > > > > > >>>>>>>> non-null, finite elements
> > > > > > > > > > > >>>>>>>> - Some people want to extend this work to
> > include
> > > > > > tensors,
> > > > > > > > > > > >>> shorter
> > > > > > > > > > > >>>>>>> floating
> > > > > > > > > > > >>>>>>>> point representations, variable lengths, NaN or
> > > > > Infinity
> > > > > > > > > > > >>> elements,
> > > > > > > > > > > >>>>> etc.
> > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > >>>>>>>> It sounds like we are nearly there, but have two
> > > > > halves
> > > > > > of
> > > > > > > > the
> > > > > > > > > > > >>>>>> community
> > > > > > > > > > > >>>>>>>> that care about different things. That just
> > means
> > > we
> > > > > > > should
> > > > > > > > > > > >> have
> > > > > > > > > > > >>>>>> multiple
> > > > > > > > > > > >>>>>>>> types. It's entirely reasonable to want a type
> > > that
> > > > > > allows
> > > > > > > > NaN
> > > > > > > > > > > >>>>> values,
> > > > > > > > > > > >>>>>>> and
> > > > > > > > > > > >>>>>>>> it's also reasonable for those of us building
> > > > > databases,
> > > > > > > > table
> > > > > > > > > > > >>>>>> standards,
> > > > > > > > > > > >>>>>>>> and vector indexes to care about guarantees that
> > > > > > elements
> > > > > > > > are
> > > > > > > > > > > >>>> finite.
> > > > > > > > > > > >>>>>>>> There's more than enough examples to show that
> > > this
> > > > > is a
> > > > > > > > > > > >>> legitimate
> > > > > > > > > > > >>>>> use
> > > > > > > > > > > >>>>>>>> case.
> > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > >>>>>>>> Since I don't see the concerns of the different
> > > use
> > > > > > cases
> > > > > > > > > > > >>> changing,
> > > > > > > > > > > >>>>> it
> > > > > > > > > > > >>>>>>>> looks like we need to find a compromise to move
> > > > > > forward. I
> > > > > > > > > > > >>> propose
> > > > > > > > > > > >>>>> that
> > > > > > > > > > > >>>>>>> we
> > > > > > > > > > > >>>>>>>> introduce a FINITE_VECTOR type and separately a
> > > > > > > > > > > >> FIXED_LENGTH_LIST
> > > > > > > > > > > >>>> or
> > > > > > > > > > > >>>>>>>> another VECTOR type that allows infinite and NaN
> > > > > values.
> > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > >>>>>>>> Can we agree on a compromise? Is there a strong
> > > reason
> > > > > > to
> > > > > > > > try
> > > > > > > > > > > >> to
> > > > > > > > > > > >>>> fit
> > > > > > > > > > > >>>>>>> these
> > > > > > > > > > > >>>>>>>> use cases into a single type? What if that means
> > > not
> > > > > > > making
> > > > > > > > > > > >>>> progress?
> > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > >>>>>>>> Ryan
> > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > >>>>>>>> On Mon, Oct 5, 2026 at 12:15 PM Philipp
> > Fischbeck
> > > via
> > > > > > dev
> > > > > > > <
> > > > > > > > > > > >>>>>>>> dev@parquet.apache.org> wrote:
> > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > >>>>>>>>> Thinking about this more, I am also fine with
> > > pulling
> > > > > > out
> > > > > > > > the
> > > > > > > > > > > >>>>>> “NaN/Inf”
> > > > > > > > > > > >>>>>>>>> requirement. Either into a type boolean (my
> > > > > > preference),
> > > > > > > or
> > > > > > > > > > > >>>> putting
> > > > > > > > > > > >>>>>> no
> > > > > > > > > > > >>>>>>>>> restrictions on the Parquet level, allowing all
> > > IEEE
> > > > > > 754
> > > > > > > > > > > >> float
> > > > > > > > > > > >>>>>> numbers.
> > > > > > > > > > > >>>>>>>> If
> > > > > > > > > > > >>>>>>>>> some top-level application wants to put
> > > additional
> > > > > > > > > > > >> constraints
> > > > > > > > > > > >>> on
> > > > > > > > > > > >>>>> the
> > > > > > > > > > > >>>>>>>> data
> > > > > > > > > > > >>>>>>>>> type, like Iceberg not allowing NaN/infinity,
> > it
> > > can
> > > > > > > still
> > > > > > > > do
> > > > > > > > > > > >>> so
> > > > > > > > > > > >>>>> one
> > > > > > > > > > > >>>>>>>> layer
> > > > > > > > > > > >>>>>>>>> above. It seems like this was the biggest point
> > > of
> > > > > > > > discussion
> > > > > > > > > > > >>>>>>> remaining.
> > > > > > > > > > > >>>>>>>>> All these options work for me, and we should
> > move
> > > > > > forward
> > > > > > > > > > > >> with
> > > > > > > > > > > >>>> this
> > > > > > > > > > > >>>>>>>>> proposal.
> > > > > > > > > > > >>>>>>>>>
> > > > > > > > > > > >>>>>>>>> Concerning additional element types like bf16,
> > > they
> > > > > can
> > > > > > > be
> > > > > > > > > > > >>>>> introduced
> > > > > > > > > > > >>>>>>> as
> > > > > > > > > > > >>>>>>>>> Parquet types (at least in the current
> > LIST-based
> > > > > spec
> > > > > > > > > > > >>> proposal).
> > > > > > > > > > > >>>>>>>>>
> > > > > > > > > > > >>>>>>>>> Concerning Will’s suggestion around
> > representing
> > > > > VECTOR
> > > > > > > as
> > > > > > > > a
> > > > > > > > > > > >>>>> variable
> > > > > > > > > > > >>>>>>>>> length list with a MIN/MAX_LENGTH metadata
> > > > > > restriction: I
> > > > > > > > > > > >> think
> > > > > > > > > > > >>>>> that
> > > > > > > > > > > >>>>>> a
> > > > > > > > > > > >>>>>>>>> dedicated logical VECTOR type still provides
> > > enough
> > > > > > > > benefits
> > > > > > > > > > > >> by
> > > > > > > > > > > >>>>>>>> describing
> > > > > > > > > > > >>>>>>>>> the semantics of the data, and allowing to
> > > capture
> > > > > > future
> > > > > > > > > > > >>> changes
> > > > > > > > > > > >>>>>>> around
> > > > > > > > > > > >>>>>>>>> statistics. In that case, the length is better
> > > > > captured
> > > > > > > as
> > > > > > > > an
> > > > > > > > > > > >>>> exact
> > > > > > > > > > > >>>>>>>> length
> > > > > > > > > > > >>>>>>>>> for this type specifically. I’m also not sure
> > > about
> > > > > > other
> > > > > > > > use
> > > > > > > > > > > >>>> cases
> > > > > > > > > > > >>>>>> for
> > > > > > > > > > > >>>>>>>>> MIN_LENGTH, so introducing a new type
> > constraint
> > > > > system
> > > > > > > > just
> > > > > > > > > > > >>> for
> > > > > > > > > > > >>>>> the
> > > > > > > > > > > >>>>>>>>> MIN_LENGTH=MAX_LENGTH case for vectors doesn’t
> > > have
> > > > > > > enough
> > > > > > > > > > > >>> value
> > > > > > > > > > > >>>> in
> > > > > > > > > > > >>>>>> my
> > > > > > > > > > > >>>>>>>>> view.
> > > > > > > > > > > >>>>>>>>>
> > > > > > > > > > > >>>>>>>>> Best,
> > > > > > > > > > > >>>>>>>>> Philipp
> > > > > > > > > > > >>>>>>>>>
> > > > > > > > > > > >>>>>>>>>
> > > > > > > > > > > >>>>>>>>> On Thu, Oct 1, 2026 at 9:00 AM Will Edwards via
> > > dev <
> > > > > > > > > > > >>>>>>>>> dev@parquet.apache.org>
> > > > > > > > > > > >>>>>>>>> wrote:
> > > > > > > > > > > >>>>>>>>>
> > > > > > > > > > > >>>>>>>>>> +1 Sebastian, a vector type suggests usage
> > > beyond
> > > > > > > > > > > >> normalized
> > > > > > > > > > > >>>>>>> embeddings
> > > > > > > > > > > >>>>>>>>> and
> > > > > > > > > > > >>>>>>>>>> engines really aren't going to be able to
> > elide
> > > all
> > > > > > the
> > > > > > > > > > > >>> checks
> > > > > > > > > > > >>>>>>> anyway.
> > > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>> Antoine, you suggested we explore metadata
> > > solutions
> > > > > > too
> > > > > > > > :)
> > > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>> Today, a database supporting CHAR(N) or
> > > VARCHAR(n)
> > > > > > can't
> > > > > > > > > > > >>>> express
> > > > > > > > > > > >>>>>> this
> > > > > > > > > > > >>>>>>>> in
> > > > > > > > > > > >>>>>>>>>> Parquet.  Engines can write opaque blobs they
> > > > > > understand
> > > > > > > > in
> > > > > > > > > > > >>> the
> > > > > > > > > > > >>>>>>>> metadata
> > > > > > > > > > > >>>>>>>>>> (e.g. Arrow does) so the extra type info
> > > > > round-trips,
> > > > > > > but
> > > > > > > > > > > >>> this
> > > > > > > > > > > >>>>>> isn't
> > > > > > > > > > > >>>>>>>>>> standardized, portable, nor interoperable.
> > > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>> So perhaps standardizing constraint metadata
> > or
> > > new
> > > > > > > Thrift
> > > > > > > > > > > >>>> fields
> > > > > > > > > > > >>>>>> (my
> > > > > > > > > > > >>>>>>>>> pref)
> > > > > > > > > > > >>>>>>>>>> can generically support VECTOR(N) whilst also
> > > being
> > > > > > more
> > > > > > > > > > > >>>> broadly
> > > > > > > > > > > >>>>>>> useful
> > > > > > > > > > > >>>>>>>>> for
> > > > > > > > > > > >>>>>>>>>> other use cases and data types?
> > > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>> Imagine metadata or Thrift fields to tag the
> > > > > > MIN_LENGTH
> > > > > > > > and
> > > > > > > > > > > >>>>>>> MAX_LENGTH
> > > > > > > > > > > >>>>>>>>> for
> > > > > > > > > > > >>>>>>>>>> Parquet's variable-length physcial types.
> > These
> > > > > would
> > > > > > > > > > > >> count
> > > > > > > > > > > >>>>>> logical
> > > > > > > > > > > >>>>>>>>> units
> > > > > > > > > > > >>>>>>>>>> (e.g. UTF8 code points for STRING) rather than
> > > > > bytes.
> > > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>> Then one engine's VECTOR(N) can be written to
> > > > > Parquet
> > > > > > as
> > > > > > > > > > > >>>> LIST(N)
> > > > > > > > > > > >>>>>> with
> > > > > > > > > > > >>>>>>>>>> MIN_LENGTH = MAX_LENGTH = N and then readers -
> > > even
> > > > > > > > > > > >> different
> > > > > > > > > > > >>>>>>> engines -
> > > > > > > > > > > >>>>>>>>> can
> > > > > > > > > > > >>>>>>>>>> pick up these constraints if they support
> > doing
> > > so.
> > > > > > We
> > > > > > > > > > > >> don't
> > > > > > > > > > > >>>>>>> actually
> > > > > > > > > > > >>>>>>>>> need
> > > > > > > > > > > >>>>>>>>>> a logical type to express a VECTOR then.
> > > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>> This also allows standard SQL's CHAR(N) and
> > > > > VARCHAR(N)
> > > > > > > to
> > > > > > > > > > > >> be
> > > > > > > > > > > >>>>>>> expressed
> > > > > > > > > > > >>>>>>>> on
> > > > > > > > > > > >>>>>>>>>> UTF8 columns etc.
> > > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>> This also seems generally useful in the future
> > > as we
> > > > > > > > > > > >> approach
> > > > > > > > > > > >>>>>>>> expressing
> > > > > > > > > > > >>>>>>>>>> other tensor data shapes and types.
> > > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>> This won't change the mechanics: the storage
> > > type
> > > > > > > doesn't
> > > > > > > > > > > >>>> enforce
> > > > > > > > > > > >>>>>>> these
> > > > > > > > > > > >>>>>>>>>> constraints and engines must check them. It's
> > > > > > previously
> > > > > > > > > > > >>>>>> established
> > > > > > > > > > > >>>>>>>> that
> > > > > > > > > > > >>>>>>>>>> performance is just a software problem and
> > > there is
> > > > > no
> > > > > > > > > > > >>>>> fundamental
> > > > > > > > > > > >>>>>>>>>> performance advantage to the fixed size types
> > in
> > > > > > > > > > > >> Dremel-like
> > > > > > > > > > > >>>>>> formats
> > > > > > > > > > > >>>>>>>> like
> > > > > > > > > > > >>>>>>>>>> Parquet.
> > > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>> Best,
> > > > > > > > > > > >>>>>>>>>> Will
> > > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>> On Wed, 30 Sept 2026 at 16:08, Sebastian
> > > Baunsgaard
> > > > > <
> > > > > > > > > > > >>>>>>>>> baunsgaard@apache.org
> > > > > > > > > > > >>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>> wrote:
> > > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>> After listening in on the last meeting
> > > Wednesday, I
> > > > > > > have
> > > > > > > > > > > >>> been
> > > > > > > > > > > >>>>>>>> thinking
> > > > > > > > > > > >>>>>>>>>>> about this quite a bit, and thought I would
> > > share
> > > > > my
> > > > > > 2
> > > > > > > > > > > >>> cents.
> > > > > > > > > > > >>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>> I suggest not requiring VECTOR elements to be
> > > > > finite,
> > > > > > > as
> > > > > > > > > > > >>>>> proposed
> > > > > > > > > > > >>>>>>> in
> > > > > > > > > > > >>>>>>>>> the
> > > > > > > > > > > >>>>>>>>>> PR
> > > > > > > > > > > >>>>>>>>>>> <
> > > https://github.com/apache/parquet-format/pull/624
> > > > > >.
> > > > > > > > > > > >>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>> While Divjot and others note the
> > > specialization for
> > > > > > > > > > > >> UTF-8,
> > > > > > > > > > > >>>>>> Parquet
> > > > > > > > > > > >>>>>>>>>> already
> > > > > > > > > > > >>>>>>>>>>> defines DOUBLE and FLOAT as IEEE encodings
> > > > > > > > > > > >>>>>>>>>>> <
> > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > >>>>
> > > > > > > > > >
> > > > > >
> > https://github.com/apache/parquet-format/blob/master/README.md#types
> > > > > > > > > > > >>>>>>>>>> .
> > > > > > > > > > > >>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>> Making NaN or +/-Inf illegal inside VECTOR
> > > would
> > > > > be a
> > > > > > > new
> > > > > > > > > > > >>>>>>> restriction
> > > > > > > > > > > >>>>>>>>> on
> > > > > > > > > > > >>>>>>>>>>> those same underlying physical types. Scalar
> > > > > > > FLOAT/DOUBLE
> > > > > > > > > > > >>>>> columns
> > > > > > > > > > > >>>>>>>> have
> > > > > > > > > > > >>>>>>>>>>> never had such a rule.
> > > > > > > > > > > >>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>> It would also not match how the systems that
> > > > > actually
> > > > > > > > > > > >>> produce
> > > > > > > > > > > >>>>>> these
> > > > > > > > > > > >>>>>>>>>> vectors
> > > > > > > > > > > >>>>>>>>>>> treat floats. Most of them use IEEE floats,
> > > e.g.
> > > > > > Spark,
> > > > > > > > > > > >>>> DuckDB,
> > > > > > > > > > > >>>>>>>> NumPy,
> > > > > > > > > > > >>>>>>>>>>> PyTorch, TensorFlow, or lower level CUDA,
> > > oneMKL,
> > > > > and
> > > > > > > > > > > >>>> OpenBLAS.
> > > > > > > > > > > >>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>> pgvector's "all elements must be finite" rule
> > > is an
> > > > > > > > > > > >> insert
> > > > > > > > > > > >>>>>> policy.
> > > > > > > > > > > >>>>>>>>> While
> > > > > > > > > > > >>>>>>>>>> I
> > > > > > > > > > > >>>>>>>>>>> think Parquet as a general-purpose format
> > > should
> > > > > > allow
> > > > > > > > > > > >> them
> > > > > > > > > > > >>>> out
> > > > > > > > > > > >>>>>> of
> > > > > > > > > > > >>>>>>>> the
> > > > > > > > > > > >>>>>>>>>> box.
> > > > > > > > > > > >>>>>>>>>>> It is a user system decision to disallow
> > > writing
> > > > > > these
> > > > > > > > > > > >>> values
> > > > > > > > > > > >>>>> or
> > > > > > > > > > > >>>>>> to
> > > > > > > > > > > >>>>>>>>> throw
> > > > > > > > > > > >>>>>>>>>>> errors when encountering them.
> > > > > > > > > > > >>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>> A type-level ban also does not keep Inf/NaN
> > > out of
> > > > > > > vector
> > > > > > > > > > > >>>> math.
> > > > > > > > > > > >>>>>>>> Finite
> > > > > > > > > > > >>>>>>>>>>> coordinates still overflow. For instance:
> > > > > magnitude =
> > > > > > > > > > > >>>>>> ||[FLOAT_MAX,
> > > > > > > > > > > >>>>>>>>>>> FLOAT_MAX]|| where every coordinate is finite
> > > > > > overflows
> > > > > > > > > > > >> to
> > > > > > > > > > > >>>>> +Inf.
> > > > > > > > > > > >>>>>>>>> PyTorch
> > > > > > > > > > > >>>>>>>>>>> documents the same overflow: tensor([1e20,
> > > > > > > 1e20]).norm()
> > > > > > > > > > > >> is
> > > > > > > > > > > >>>> inf
> > > > > > > > > > > >>>>>> in
> > > > > > > > > > > >>>>>>>>> fp32.
> > > > > > > > > > > >>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>> A generated MAGNITUDE(vec) column would then
> > > be Inf
> > > > > > > while
> > > > > > > > > > > >>> the
> > > > > > > > > > > >>>>>>> VECTOR
> > > > > > > > > > > >>>>>>>>>> itself
> > > > > > > > > > > >>>>>>>>>>> was required to be finite. The ban only
> > forbids
> > > > > > storing
> > > > > > > > > > > >>>>> specials
> > > > > > > > > > > >>>>>> in
> > > > > > > > > > > >>>>>>>> the
> > > > > > > > > > > >>>>>>>>>>> coordinates, and does not prevent even simple
> > > > > > > downstream
> > > > > > > > > > > >>>>>>> computations
> > > > > > > > > > > >>>>>>>>>> from
> > > > > > > > > > > >>>>>>>>>>> producing them.
> > > > > > > > > > > >>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>> I suggest keeping VECTOR as the type and add
> > an
> > > > > > > optional
> > > > > > > > > > > >>> bool
> > > > > > > > > > > >>>>>>>>>>> VectorType.finite (default false). Engines
> > that
> > > > > want
> > > > > > > > > > > >>>>>> pgvector-like
> > > > > > > > > > > >>>>>>>>>>> rejection can require the flag = true.
> > Engines
> > > that
> > > > > > use
> > > > > > > > > > > >>>> NumPy /
> > > > > > > > > > > >>>>>>>>> PyTorch /
> > > > > > > > > > > >>>>>>>>>>> Spark arrays can keep representable IEEE
> > > values.
> > > > > For
> > > > > > > the
> > > > > > > > > > > >>> same
> > > > > > > > > > > >>>>>>>> reason, I
> > > > > > > > > > > >>>>>>>>>> do
> > > > > > > > > > > >>>>>>>>>>> not favor adding yet another type called
> > > > > > FINITE_VECTOR.
> > > > > > > > > > > >>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>> Best,
> > > > > > > > > > > >>>>>>>>>>> Sebastian
> > > > > > > > > > > >>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>> On Tue, Sep 29, 2026 at 2:03 PM Divjot Arora
> > > via
> > > > > dev
> > > > > > <
> > > > > > > > > > > >>>>>>>>>>> dev@parquet.apache.org>
> > > > > > > > > > > >>>>>>>>>>> wrote:
> > > > > > > > > > > >>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>> I agree with moving forward with the
> > proposal
> > > and
> > > > > > > > > > > >>> renaming
> > > > > > > > > > > >>>>>> VECTOR
> > > > > > > > > > > >>>>>>>> to
> > > > > > > > > > > >>>>>>>>>>>> FINITE_VECTOR or something
> > > > > > > > > > > >>>>>>>>>>>> similar if others feel that better
> > > communicates
> > > > > its
> > > > > > > > > > > >>>> intent. I
> > > > > > > > > > > >>>>>> see
> > > > > > > > > > > >>>>>>>>> value
> > > > > > > > > > > >>>>>>>>>>> in
> > > > > > > > > > > >>>>>>>>>>>> having a vector type that guarantees
> > > > > > > > > > > >>>>>>>>>>>> its values are mathematically-correct
> > vectors
> > > for
> > > > > a
> > > > > > > few
> > > > > > > > > > > >>>>>> reasons:
> > > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>> 1. As others have mentioned, this is similar
> > > to
> > > > > the
> > > > > > > > > > > >>> STRING
> > > > > > > > > > > >>>>>>> logical
> > > > > > > > > > > >>>>>>>>> type
> > > > > > > > > > > >>>>>>>>>>>> mandating UTF-8.
> > > > > > > > > > > >>>>>>>>>>>> 2. It is not possible to rely on statistics
> > > for
> > > > > > > > > > > >> detecting
> > > > > > > > > > > >>>> NaN
> > > > > > > > > > > >>>>>> or
> > > > > > > > > > > >>>>>>>>> +-inf
> > > > > > > > > > > >>>>>>>>>>>> values as all statistics are optional.
> > > > > > > > > > > >>>>>>>>>>>> To my knowledge, there is no precedent for
> > > > > requiring
> > > > > > > > > > > >> that
> > > > > > > > > > > >>>>>> writers
> > > > > > > > > > > >>>>>>>>> emit
> > > > > > > > > > > >>>>>>>>>>>> stats for specific logical types.
> > > > > > > > > > > >>>>>>>>>>>> 3. Without statistics, NaN/inf values could
> > > flow
> > > > > > > > > > > >> through
> > > > > > > > > > > >>>> the
> > > > > > > > > > > >>>>>>> engine
> > > > > > > > > > > >>>>>>>>> and
> > > > > > > > > > > >>>>>>>>>>>> block them for specific functionality.
> > > > > > > > > > > >>>>>>>>>>>> This seems to be Lance's behavior. However,
> > I
> > > > > don't
> > > > > > > > > > > >> think
> > > > > > > > > > > >>>>> this
> > > > > > > > > > > >>>>>> is
> > > > > > > > > > > >>>>>>>> as
> > > > > > > > > > > >>>>>>>>>> easy
> > > > > > > > > > > >>>>>>>>>>>> as it sounds and Lance has run
> > > > > > > > > > > >>>>>>>>>>>> into several bugs due to this ([1] [2] for
> > > > > > examples).
> > > > > > > > > > > >>> Their
> > > > > > > > > > > >>>>>>> Python
> > > > > > > > > > > >>>>>>>>> [3]
> > > > > > > > > > > >>>>>>>>>>> and
> > > > > > > > > > > >>>>>>>>>>>> Rust [4] APIs default to erroring on
> > > > > > > > > > > >>>>>>>>>>>> vectors that contain NaN values.
> > > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>> I agree that Parquet is a general-purpose
> > > format,
> > > > > > but
> > > > > > > I
> > > > > > > > > > > >>>> don't
> > > > > > > > > > > >>>>>>> think
> > > > > > > > > > > >>>>>>>>>> this
> > > > > > > > > > > >>>>>>>>>>>> implies the lack of domain-specific types.
> > > > > > > > > > > >>>>>>>>>>>> Instead, this broadens the use cases where
> > > Parquet
> > > > > > > > > > > >> would
> > > > > > > > > > > >>> be
> > > > > > > > > > > >>>>>>>>> considered
> > > > > > > > > > > >>>>>>>>>> a
> > > > > > > > > > > >>>>>>>>>>>> good fit. I also fully agree with
> > > > > > > > > > > >>>>>>>>>>>> continuing work on a general fixed-size list
> > > type
> > > > > if
> > > > > > > we
> > > > > > > > > > > >>>>> believe
> > > > > > > > > > > >>>>>>>> there
> > > > > > > > > > > >>>>>>>>>> are
> > > > > > > > > > > >>>>>>>>>>>> non-vector use cases that would benefit
> > > > > > > > > > > >>>>>>>>>>>> from such guarantees at the storage layer.
> > > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>> [1]
> > > > > https://github.com/lance-format/lance/pull/3648
> > > > > > > > > > > >>>>>>>>>>>> [2]
> > > > > > https://github.com/lance-format/lance/issues/8607
> > > > > > > > > > > >>> (see
> > > > > > > > > > > >>>>>>> point 2
> > > > > > > > > > > >>>>>>>>> in
> > > > > > > > > > > >>>>>>>>>>>> "root
> > > > > > > > > > > >>>>>>>>>>>> cause")
> > > > > > > > > > > >>>>>>>>>>>> [3]
> > > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>
> > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > >>>>>>>
> > > > > > > > > > > >>>>>>
> > > > > > > > > > > >>>>>
> > > > > > > > > > > >>>>
> > > > > > > > > > > >>>
> > > > > > > > > > > >>
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > >
> > https://github.com/lancedb/lancedb/blob/0327618a62483feb6dca2336549845e5887cd48f/python/python/lancedb/table.py#L4807-L4911
> > > > > > > > > > > >>>>>>>>>>>> [4]
> > > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>
> > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > >>>>>>>
> > > > > > > > > > > >>>>>>
> > > > > > > > > > > >>>>>
> > > > > > > > > > > >>>>
> > > > > > > > > > > >>>
> > > > > > > > > > > >>
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > >
> > https://github.com/lancedb/lancedb/blob/0327618a62483feb6dca2336549845e5887cd48f/rust/lancedb/src/table/add_data.rs#L105-L110
> > > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>> On Mon, Sep 28, 2026 at 6:19 PM Philipp
> > > Fischbeck
> > > > > > via
> > > > > > > > > > > >>> dev <
> > > > > > > > > > > >>>>>>>>>>>> dev@parquet.apache.org> wrote:
> > > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>> I also favor moving forward with the
> > > proposal as
> > > > > > is.
> > > > > > > > > > > >> I
> > > > > > > > > > > >>>> see
> > > > > > > > > > > >>>>>>> strong
> > > > > > > > > > > >>>>>>>>>> value
> > > > > > > > > > > >>>>>>>>>>>> and
> > > > > > > > > > > >>>>>>>>>>>>> use cases for the numeric vector type, and
> > I
> > > > > agree
> > > > > > > > > > > >> with
> > > > > > > > > > > >>>> the
> > > > > > > > > > > >>>>>>>>>>> restrictions
> > > > > > > > > > > >>>>>>>>>>>>> around NaN/Inf values and element types in
> > > > > general.
> > > > > > > > > > > >>>>> Precedent
> > > > > > > > > > > >>>>>>>>> exists
> > > > > > > > > > > >>>>>>>>>> in
> > > > > > > > > > > >>>>>>>>>>>>> other types that restrict which physical
> > type
> > > > > > values
> > > > > > > > > > > >>> are
> > > > > > > > > > > >>>>>>>> logically
> > > > > > > > > > > >>>>>>>>>>> valid
> > > > > > > > > > > >>>>>>>>>>>>> (e.g., the geospatial types needing to be
> > > > > > > > > > > >> WKB-encoded).
> > > > > > > > > > > >>>> A
> > > > > > > > > > > >>>>>>>> general
> > > > > > > > > > > >>>>>>>>>>>>> fixed-size list also has value, but I think
> > > it
> > > > > > makes
> > > > > > > > > > > >>>> sense
> > > > > > > > > > > >>>>> to
> > > > > > > > > > > >>>>>>>>>> separate
> > > > > > > > > > > >>>>>>>>>>>> the
> > > > > > > > > > > >>>>>>>>>>>>> two types. The semantics are relevant, and
> > I
> > > can
> > > > > > see
> > > > > > > > > > > >>> how
> > > > > > > > > > > >>>>> the
> > > > > > > > > > > >>>>>>> two
> > > > > > > > > > > >>>>>>>>>> types
> > > > > > > > > > > >>>>>>>>>>>>> might diverge further in the future
> > regarding
> > > > > > > > > > > >>> statistics
> > > > > > > > > > > >>>> or
> > > > > > > > > > > >>>>>>>>>> encodings.
> > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>> I would be okay with renaming the vector
> > > type,
> > > > > but
> > > > > > it
> > > > > > > > > > > >>>> seems
> > > > > > > > > > > >>>>>>> like
> > > > > > > > > > > >>>>>>>>> the
> > > > > > > > > > > >>>>>>>>>>>> VECTOR
> > > > > > > > > > > >>>>>>>>>>>>> name is already used across other engines
> > > > > mentioned
> > > > > > > > > > > >> in
> > > > > > > > > > > >>>> this
> > > > > > > > > > > >>>>>>>> thread.
> > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>> Best,
> > > > > > > > > > > >>>>>>>>>>>>> Philipp
> > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>> On Sat, Sep 26, 2026 at 4:37 PM Gang Wu <
> > > > > > > > > > > >>>> ustcwg@gmail.com>
> > > > > > > > > > > >>>>>>>> wrote:
> > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>>> To separate orthogonal concerns, I feel we
> > > > > should
> > > > > > > > > > > >>> treat
> > > > > > > > > > > >>>>> the
> > > > > > > > > > > >>>>>>>>> general
> > > > > > > > > > > >>>>>>>>>>>>>> FixedSizeList as the underlying physical
> > > layout,
> > > > > > > > > > > >> and
> > > > > > > > > > > >>>>>> separate
> > > > > > > > > > > >>>>>>>> it
> > > > > > > > > > > >>>>>>>>>> from
> > > > > > > > > > > >>>>>>>>>>>> the
> > > > > > > > > > > >>>>>>>>>>>>>> discussion of a specialized VECTOR (or
> > > broader
> > > > > ML
> > > > > > > > > > > >>>> tensor)
> > > > > > > > > > > >>>>>>>> logical
> > > > > > > > > > > >>>>>>>>>>> type.
> > > > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>>> Currently, the immediate need is to
> > define a
> > > > > > > > > > > >>>> specialized
> > > > > > > > > > > >>>>>>> VECTOR
> > > > > > > > > > > >>>>>>>>>> type
> > > > > > > > > > > >>>>>>>>>>> so
> > > > > > > > > > > >>>>>>>>>>>>>> that different engines and formats have a
> > > common
> > > > > > > > > > > >>>> semantic
> > > > > > > > > > > >>>>>>>>> contract
> > > > > > > > > > > >>>>>>>>>>> for
> > > > > > > > > > > >>>>>>>>>>>>>> interoperability.
> > > > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>>> If the main point of contention is just
> > the
> > > > > > > > > > > >>>> NaN/Inf/null
> > > > > > > > > > > >>>>>>>>>> limitations,
> > > > > > > > > > > >>>>>>>>>>>>> maybe
> > > > > > > > > > > >>>>>>>>>>>>>> we can resolve this by adding a
> > > qualification to
> > > > > > > > > > > >> the
> > > > > > > > > > > >>>> name
> > > > > > > > > > > >>>>>>> (like
> > > > > > > > > > > >>>>>>>>>>> Andrew
> > > > > > > > > > > >>>>>>>>>>>>>> suggested, e.g., VECTOR_EMBEDDING or
> > > > > > FINITE_VECTOR)
> > > > > > > > > > > >>> to
> > > > > > > > > > > >>>>>> avoid
> > > > > > > > > > > >>>>>>>>>>> confusion
> > > > > > > > > > > >>>>>>>>>>>>> with
> > > > > > > > > > > >>>>>>>>>>>>>> broader ML use cases. Alternatively, we
> > > could
> > > > > > > > > > > >>>> introduce a
> > > > > > > > > > > >>>>>>>> boolean
> > > > > > > > > > > >>>>>>>>>>> flag
> > > > > > > > > > > >>>>>>>>>>>>>> (e.g., optional bool VectorType.finite) as
> > > > > > proposed
> > > > > > > > > > > >>>>>> earlier.
> > > > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>>> On Fri, Sep 25, 2026 at 3:52 AM Andrew
> > Lamb
> > > <
> > > > > > > > > > > >>>>>>>>>> andrewlamb11@gmail.com>
> > > > > > > > > > > >>>>>>>>>>>>>> wrote:
> > > > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>>>> I am a little confused about the points
> > of
> > > > > > > > > > > >>>> disagreement
> > > > > > > > > > > >>>>>>> here.
> > > > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>>>> Is it that the name `VECTOR` is too
> > > general for
> > > > > > > > > > > >> the
> > > > > > > > > > > >>>>>>> specific
> > > > > > > > > > > >>>>>>>>> use
> > > > > > > > > > > >>>>>>>>>>> case
> > > > > > > > > > > >>>>>>>>>>>>> of
> > > > > > > > > > > >>>>>>>>>>>>>>> fixed length arrays with non-finite
> > values
> > > used
> > > > > > > > > > > >> for
> > > > > > > > > > > >>>>>> vector
> > > > > > > > > > > >>>>>>>>>>>> embeddings?
> > > > > > > > > > > >>>>>>>>>>>>>> If
> > > > > > > > > > > >>>>>>>>>>>>>>> so, maybe we could add a qualification to
> > > the
> > > > > > > > > > > >> name
> > > > > > > > > > > >>>>>>>>>> (FINITE_VECTOR?
> > > > > > > > > > > >>>>>>>>>>>>>>> FEATURE_VECTOR? VECTOR_EMBEDDING?)
> > > > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>>>> It seems like there is precedent for
> > > defining
> > > > > > > > > > > >>> logical
> > > > > > > > > > > >>>>>> types
> > > > > > > > > > > >>>>>>>>> that
> > > > > > > > > > > >>>>>>>>>>>>> specify
> > > > > > > > > > > >>>>>>>>>>>>>>> certain physical types and restrict their
> > > > > > > > > > > >> contents
> > > > > > > > > > > >>> to
> > > > > > > > > > > >>>>>> help
> > > > > > > > > > > >>>>>>>>>> readers
> > > > > > > > > > > >>>>>>>>>>>>>>> interpret meaning, for example string[1]
> > or
> > > > > JSON;
> > > > > > > > > > > >>>>>> Defining
> > > > > > > > > > > >>>>>>>> one
> > > > > > > > > > > >>>>>>>>>> for
> > > > > > > > > > > >>>>>>>>>>>>>> vector
> > > > > > > > > > > >>>>>>>>>>>>>>> embeddings seems reasonable enough to me.
> > > > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>>>> Andrew
> > > > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>>>> [1]:
> > > > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>
> > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > >>>>>>>
> > > > > > > > > > > >>>>>>
> > > > > > > > > > > >>>>>
> > > > > > > > > > > >>>>
> > > > > > > > > > > >>>
> > > > > > > > > > > >>
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > >
> > https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#string
> > > > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>>>> On Thu, Sep 24, 2026 at 3:34 PM Daniel
> > > Weeks <
> > > > > > > > > > > >>>>>>>>> dweeks@apache.org>
> > > > > > > > > > > >>>>>>>>>>>>> wrote:
> > > > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>>>>>> Well, Parquet is not a single-purpose
> > > format.
> > > > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>>>>>> Vector stores can choose whatever
> > policy
> > > fits
> > > > > > > > > > > >>>> their
> > > > > > > > > > > >>>>>>>> usage,
> > > > > > > > > > > >>>>>>>>>> but
> > > > > > > > > > > >>>>>>>>>>>>> we're
> > > > > > > > > > > >>>>>>>>>>>>>>>>> talking about a general-purpose file
> > > format
> > > > > > > > > > > >>> that
> > > > > > > > > > > >>>>> aims
> > > > > > > > > > > >>>>>>> to
> > > > > > > > > > > >>>>>>>> be
> > > > > > > > > > > >>>>>>>>>>>> broadly
> > > > > > > > > > > >>>>>>>>>>>>>>>>> applicable.
> > > > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>>>>> I agree that the format is intended to
> > be
> > > > > > > > > > > >> general
> > > > > > > > > > > >>>>>>> purpose,
> > > > > > > > > > > >>>>>>>>> but
> > > > > > > > > > > >>>>>>>>>>>> types
> > > > > > > > > > > >>>>>>>>>>>>>>> within
> > > > > > > > > > > >>>>>>>>>>>>>>>> the format define specific
> > representation
> > > and
> > > > > > > > > > > >>>>> semantics
> > > > > > > > > > > >>>>>>> to
> > > > > > > > > > > >>>>>>>>>>> achieve
> > > > > > > > > > > >>>>>>>>>>>>> that
> > > > > > > > > > > >>>>>>>>>>>>>>>> goal.
> > > > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>>>>>> "Vector" has a more general sense than
> > > ML and
> > > > > > > > > > > >>>> it's
> > > > > > > > > > > >>>>>>> quite
> > > > > > > > > > > >>>>>>>>>>> trivial
> > > > > > > > > > > >>>>>>>>>>>> to
> > > > > > > > > > > >>>>>>>>>>>>>>>>> produce vectors with non-finite values
> > > using
> > > > > > > > > > > >>> e.g.
> > > > > > > > > > > >>>>>> NumPy
> > > > > > > > > > > >>>>>>>> or
> > > > > > > > > > > >>>>>>>>>>> Pandas
> > > > > > > > > > > >>>>>>>>>>>>> (or
> > > > > > > > > > > >>>>>>>>>>>>>>>>> any other piece of dataframe or
> > > scientific
> > > > > > > > > > > >>>>> computing
> > > > > > > > > > > >>>>>>>>>> software).
> > > > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>>>>> The justification for restricting
> > > non-finite
> > > > > > > > > > > >>>> values
> > > > > > > > > > > >>>>> is
> > > > > > > > > > > >>>>>>>>>> directly
> > > > > > > > > > > >>>>>>>>>>>>>> related
> > > > > > > > > > > >>>>>>>>>>>>>>> to
> > > > > > > > > > > >>>>>>>>>>>>>>>> the semantic meaning of a vector.
> > Vectors
> > > > > have
> > > > > > > > > > > >>>>>> specific
> > > > > > > > > > > >>>>>>>>>>>>>> characteristics
> > > > > > > > > > > >>>>>>>>>>>>>>>> (dimension, magnitude, direction, etc).
> > > > > > > > > > > >>> Non-finite
> > > > > > > > > > > >>>>>>> values
> > > > > > > > > > > >>>>>>>>> are
> > > > > > > > > > > >>>>>>>>>>>>> invalid
> > > > > > > > > > > >>>>>>>>>>>>>>> and
> > > > > > > > > > > >>>>>>>>>>>>>>>> break those characteristics.  I'd liken
> > > it to
> > > > > > > > > > > >>>> storing
> > > > > > > > > > > >>>>>>>>> non-utf8
> > > > > > > > > > > >>>>>>>>>>>> values
> > > > > > > > > > > >>>>>>>>>>>>>> in
> > > > > > > > > > > >>>>>>>>>>>>>>> a
> > > > > > > > > > > >>>>>>>>>>>>>>>> string and still calling it a string.
> > > > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>>>>> That also differentiates a vector from a
> > > fixed
> > > > > > > > > > > >>>> length
> > > > > > > > > > > >>>>>>> list,
> > > > > > > > > > > >>>>>>>>>> which
> > > > > > > > > > > >>>>>>>>>>>> is
> > > > > > > > > > > >>>>>>>>>>>>>> more
> > > > > > > > > > > >>>>>>>>>>>>>>>> general and does not carry the same
> > > semantic
> > > > > > > > > > > >>>> meaning.
> > > > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>>>>> I also don't think we need to bundle all
> > > > > > > > > > > >>> similarly
> > > > > > > > > > > >>>>>> shaped
> > > > > > > > > > > >>>>>>>> use
> > > > > > > > > > > >>>>>>>>>>> cases
> > > > > > > > > > > >>>>>>>>>>>>>> into
> > > > > > > > > > > >>>>>>>>>>>>>>>> the same solution (e.g. flxed size
> > list).
> > > > > > > > > > > >> Vector
> > > > > > > > > > > >>>> is
> > > > > > > > > > > >>>>> a
> > > > > > > > > > > >>>>>>> well
> > > > > > > > > > > >>>>>>>>>>>>> established
> > > > > > > > > > > >>>>>>>>>>>>>>>> data type at this point and aligns with
> > > the
> > > > > > > > > > > >>> general
> > > > > > > > > > > >>>>>>> purpose
> > > > > > > > > > > >>>>>>>>>>> nature
> > > > > > > > > > > >>>>>>>>>>>> of
> > > > > > > > > > > >>>>>>>>>>>>>> the
> > > > > > > > > > > >>>>>>>>>>>>>>>> format (many other formats and databases
> > > > > > > > > > > >> natively
> > > > > > > > > > > >>>>>> support
> > > > > > > > > > > >>>>>>>>>> this).
> > > > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>>>>> -Dan
> > > > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>>>>> On Thu, Sep 24, 2026 at 11:55 AM Antoine
> > > > > > > > > > > >> Pitrou <
> > > > > > > > > > > >>>>>>>>>>>> antoine@python.org>
> > > > > > > > > > > >>>>>>>>>>>>>>>> wrote:
> > > > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>>>>>> Le 24/09/2026 à 20:11, Russell Spitzer
> > a
> > > > > > > > > > > >> écrit
> > > > > > > > > > > >>> :
> > > > > > > > > > > >>>>>>>>>>>>>>>>>> I think something I haven't seen yet
> > is
> > > the
> > > > > > > > > > > >>>> clear
> > > > > > > > > > > >>>>>>>>> use-case
> > > > > > > > > > > >>>>>>>>>>> for
> > > > > > > > > > > >>>>>>>>>>>>>>>> non-finite
> > > > > > > > > > > >>>>>>>>>>>>>>>>>> values within a vector type. There has
> > > > > > > > > > > >> been a
> > > > > > > > > > > >>>> lot
> > > > > > > > > > > >>>>>> of
> > > > > > > > > > > >>>>>>>>>>>> theoretical
> > > > > > > > > > > >>>>>>>>>>>>>>>>> discussion
> > > > > > > > > > > >>>>>>>>>>>>>>>>>> here about use cases but what toolings
> > > > > > > > > > > >>>> currently
> > > > > > > > > > > >>>>>>>> produce
> > > > > > > > > > > >>>>>>>>>>> these
> > > > > > > > > > > >>>>>>>>>>>>>>> vectors
> > > > > > > > > > > >>>>>>>>>>>>>>>>> and
> > > > > > > > > > > >>>>>>>>>>>>>>>>>> how do they function in downstream
> > > engines?
> > > > > > > > > > > >>> I'd
> > > > > > > > > > > >>>>>> like
> > > > > > > > > > > >>>>>>> to
> > > > > > > > > > > >>>>>>>>>> limit
> > > > > > > > > > > >>>>>>>>>>>>>>>> introducing
> > > > > > > > > > > >>>>>>>>>>>>>>>>>> types that have a wide range of values
> > > > > > > > > > > >> which
> > > > > > > > > > > >>>> are
> > > > > > > > > > > >>>>>>> simply
> > > > > > > > > > > >>>>>>>>> are
> > > > > > > > > > > >>>>>>>>>>>>> failure
> > > > > > > > > > > >>>>>>>>>>>>>>>> modes
> > > > > > > > > > > >>>>>>>>>>>>>>>>>> of different flavors
> > > > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>>>>>> "Vector" has a more general sense than
> > > ML and
> > > > > > > > > > > >>>> it's
> > > > > > > > > > > >>>>>>> quite
> > > > > > > > > > > >>>>>>>>>>> trivial
> > > > > > > > > > > >>>>>>>>>>>> to
> > > > > > > > > > > >>>>>>>>>>>>>>>>> produce vectors with non-finite values
> > > using
> > > > > > > > > > > >>> e.g.
> > > > > > > > > > > >>>>>> NumPy
> > > > > > > > > > > >>>>>>>> or
> > > > > > > > > > > >>>>>>>>>>> Pandas
> > > > > > > > > > > >>>>>>>>>>>>> (or
> > > > > > > > > > > >>>>>>>>>>>>>>>>> any other piece of dataframe or
> > > scientific
> > > > > > > > > > > >>>>> computing
> > > > > > > > > > > >>>>>>>>>> software).
> > > > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>>>>>> It's not like the general concept of a
> > > vector
> > > > > > > > > > > >>> is
> > > > > > > > > > > >>>> a
> > > > > > > > > > > >>>>>> IETF
> > > > > > > > > > > >>>>>>>>>>> standard
> > > > > > > > > > > >>>>>>>>>>>> or
> > > > > > > > > > > >>>>>>>>>>>>>>>>> anything, so analogies with JSON or
> > UTF8
> > > > > > > > > > > >> (where
> > > > > > > > > > > >>>> the
> > > > > > > > > > > >>>>>>>> values
> > > > > > > > > > > >>>>>>>>>> are
> > > > > > > > > > > >>>>>>>>>>>>>> expected
> > > > > > > > > > > >>>>>>>>>>>>>>>>> to be valid according to the respective
> > > > > > > > > > > >> specs)
> > > > > > > > > > > >>> do
> > > > > > > > > > > >>>>> not
> > > > > > > > > > > >>>>>>>>> apply.
> > > > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>>>>>> This discussion would IMHO have a
> > better
> > > > > > > > > > > >> chance
> > > > > > > > > > > >>>> of
> > > > > > > > > > > >>>>>>>>> converging
> > > > > > > > > > > >>>>>>>>>>> if
> > > > > > > > > > > >>>>>>>>>>>> we
> > > > > > > > > > > >>>>>>>>>>>>>>>>> separated the two orthogonal concerns
> > of:
> > > > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>>>>>> 1. representing fixed-length
> > list/vector
> > > data
> > > > > > > > > > > >>> in
> > > > > > > > > > > >>>>>>> Parquet
> > > > > > > > > > > >>>>>>>>>> (with
> > > > > > > > > > > >>>>>>>>>>> a
> > > > > > > > > > > >>>>>>>>>>>>>>>>> potential limitation to non-nested
> > types)
> > > > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>>>>>> 2. representing the information that a
> > > given
> > > > > > > > > > > >>>> column
> > > > > > > > > > > >>>>>> (or
> > > > > > > > > > > >>>>>>>>>> column
> > > > > > > > > > > >>>>>>>>>>>>> chunk,
> > > > > > > > > > > >>>>>>>>>>>>>>>>> etc.) does not contain non-finite
> > values
> > > > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>>>>>> ... both of which are useful on their
> > own
> > > > > > > > > > > >>> right,
> > > > > > > > > > > >>>>> but
> > > > > > > > > > > >>>>>> do
> > > > > > > > > > > >>>>>>>> not
> > > > > > > > > > > >>>>>>>>>>> have
> > > > > > > > > > > >>>>>>>>>>>> to
> > > > > > > > > > > >>>>>>>>>>>>>> be
> > > > > > > > > > > >>>>>>>>>>>>>>>>> conflated and bundled together.
> > > > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>>>>>> Regards
> > > > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>>>>>> Antoine.
> > > > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>
> > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > >>>>>>>
> > > > > > > > > > > >>>>>>
> > > > > > > > > > > >>>>>
> > > > > > > > > > > >>>>
> > > > > > > > > > > >>>
> > > > > > > > > > > >>
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > >
> >
>

Re: [DISCUSS] Numeric Vector type: physical representation (outcome of focused call)

[Will Edwards via dev <de...@parquet.apache.org>]

Hi Ryan,

Thanks for the detailed replies. Let me restate my position more directly,
since I don't think I made it clear.

I'd like to put the vector type on hold in Parquet until there's a general
proposal for ML tensors and sparse, packed or quantised representations.
That's where a new type would let engines do interoperable things they
can't do today.

My earlier question was what VECTOR makes possible that isn't possible
today. The answers so far describe what the annotation asserts: that the
data is a vector, that it has a fixed length, and that its values are
finite and non-null. I'm asking for a workload that can't be done on
LIST<FLOAT> now and would work with VECTOR. Jan asked the same question,
and I don't think either of us has seen one yet.

For embeddings, LIST<FLOAT> is what is already widely used today, and it
works. My compatibility concern is about writers. A tool that doesn't know
VECTOR rewrites the file as plain LIST<FLOAT>. If engines then require the
annotation, that data stops working. If they don't require it, the
annotation adds little. A world where VECTOR exists and plain LIST<FLOAT>
still works fully for ANN etc is a world that doesn't need VECTOR.

On optimisations, Dan framed this as semantics rather than performance, but
you've listed decoding optimisations. In my experience readers can remove
the cost of variable-length lists without any format change, and I'm happy
to share the recipes with anyone interested. If fixed length is still the
real gain, a fixed-size list gives it to everyone, NaN or not.

If table formats need a finite-vector contract, I think the table layer is
a better home than the file format. An Iceberg type would apply equally to
Parquet, Avro and ORC. I'd still be wary of Iceberg rejecting untagged
columns, but it's a better place for that debate than Parquet.

Best,
Will

On Thu, 22 Oct 2026 at 02:00, Ryan Blue <bl...@apache.org> wrote:

> Thanks, Curt. I wasn't aware of that behavior in MySQL. I was trying to
> learn more and I don't see it in documentation either. Do you have any more
> information about how MySQL handles this? One page makes it sound like it
> may be platform dependent
> <https://dev.mysql.com/doc/refman/26.7/en/problems-with-float.html>, where
> some values produce "inf" and some produce "0".
>
> My original comment was referring to databases that allow users to specify
> types that exclude non-finite values, which is what we would need in order
> to support a finite float in table formats since we already support the
> full range of values. If you know of any cases like that rather than just
> limiting values, I'm curious. Thanks!
>
> Ryan
>
> On Wed, Oct 21, 2026 at 3:48 PM Curt Hagenlocher <cu...@hagenlocher.org>
> wrote:
>
> > > I'm not aware of databases with finite floating point types.
> >
> > Microsoft SQL Server, MySQL and SAP ASE (née Sybase) do not allow
> > storing non-finite floating-point values in their binary
> > floating-point columns. They don't have a separate type which does
> > allow this.
> >
> > On Wed, Oct 21, 2026 at 3:30 PM Ryan Blue <bl...@apache.org> wrote:
> > >
> > > Jan,
> > >
> > > I agree that we _could_ model vectors as an orthogonal finiteness
> > property
> > > for elements in a fixed-length list. I don't know how useful that would
> > be.
> > > I've not had anyone ask for that, and I'm not aware of databases with
> > > finite floating point types. I think it makes more sense to move
> forward
> > > with the existing FINITE_VECTOR that is proposed.
> > >
> > > For the logical type argument, it sounds like the issue is that you
> want
> > to
> > > fall back to readers treating vectors as LIST. I don't see a good way
> to
> > > make that happen, but it also isn't needed by the table format use case
> > > because we are introducing this as a new type. Do you have a use case
> > where
> > > you are going to mix FINITE_VECTOR with LIST files and this will cause
> a
> > > problem?
> > >
> > > I don't think that we've had an expectation for logical types in the
> past
> > > that they produce a particular in-memory representation. Maybe we could
> > > allow setting both `LIST` and `FINITE_VECTOR` logical types at the same
> > > time? This kind of compatibility hasn't been a priority as far as I'm
> > > aware, but if it is important to you we can consider it.
> > >
> > > Ryan
> > >
> > > On Wed, Oct 21, 2026 at 1:13 PM Jan Finis <jp...@gmail.com> wrote:
> > >
> > > >  assuming what will be proposed is a new logical type FINITE_VECTOR
> > that
> > > > annotates the same column that would have been annotated as LIST
> > before:
> > > >
> > > > What I find somewhat suboptimal of FINITE_VECTOR is that finiteness
> is
> > part
> > > > of the vector type. Isn't the finiteness rather a property of the
> > contained
> > > > floats? Such a modeling would be more composable, as it would for
> > example
> > > > also allow to have normal (non-vector) float columns that are
> > restricted to
> > > > be finite. So, if we do think that we want finiteness in the type
> > system, I
> > > > would rather cut it into two parts: A fixed size vector concept, and
> a
> > new
> > > > way of labeling float columns finite. The vector type would then just
> > boil
> > > > down to be a fixed size list / we could just give the list type an
> > optional
> > > > fixed size field. Then, the column contained within it would be a
> float
> > > > tagged as finite.
> > > >
> > > > It was mentioned that a logical type annotating a list of float would
> > be
> > > > forward compatible, but - depending on the modeling - that is not
> > > > necessarily true: as logical types in Parquet do not support
> > subtyping, a
> > > > column cannot be a list of float and a (finite) vector at the same
> > time,
> > > > but only one of the two. It depends on how the type will be modeled
> in
> > > > thrift in the end, but if it is a new union member of `LogicalType`,
> > then
> > > > legacy engines will not understand that such a vector can also just
> be
> > > > interpreted as a list of floats. They will just see a nested column
> > type of
> > > > a logical type they don't understand, so they will likely reject
> > reading
> > > > this column (at least that's what the engine I'm maintaining would be
> > > > doing).
> > > >
> > > > In contrast, if we just give the list type an optional fixed size and
> > float
> > > > an optional finiteness, then the resulting thrift is actually fully
> > forward
> > > > compatible: Legacy engines will just ignore the fixed length and
> treat
> > the
> > > > list as variable length and they will ignore the finiteness and treat
> > the
> > > > element type as float.
> > > >
> > > > (A side note, which most people here may already be aware of, but
> it's
> > good
> > > > to repeat it: The not-NULL property doesn't need new type support at
> > all.
> > > > The normal type system of list already allows setting the contained
> > element
> > > > type to be optional or required. If it is set to be required, then
> > NULLs
> > > > are not allowed.)
> > > >
> > > > Cheers,
> > > > Jan
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > Am Di., 20. Okt. 2026 um 20:37 Uhr schrieb Russell Spitzer <
> > > > russell.spitzer@gmail.com>:
> > > >
> > > > > I support proceeding with the two types. I honestly think the cost
> of
> > > > > adding new logical types is overestimated. If we have as many
> > interested
> > > > > parties as have appeared in this thread we already know there is a
> > large
> > > > > group of engineers ready to do the work and engines ready to
> consume
> > (and
> > > > > produce) the type. The fact that implementations currently use
> > > > heuristics,
> > > > > sampling (I loved Gunnar's talk on his hardwood work on this at CoC
> > > > > Glasgow) or custom annotations to achieve the same performance
> > benefits
> > > > > makes me think we should just create the type and move on.
> > > > >
> > > > > On Mon, Oct 19, 2026 at 7:21 PM Ryan Blue <bl...@apache.org> wrote:
> > > > >
> > > > > > An annotation on list<float> is useful because:
> > > > > > - We want to correctly annotate what is being stored
> > > > > > - We want to unlock optimizations when each non-null list has the
> > same
> > > > > > number of elements
> > > > > > - We want to have writer guarantees
> > > > > >
> > > > > > I don't think that logical type compatibility is a good argument.
> > We
> > > > want
> > > > > > to avoid a case where we have to treat all binary like a string.
> > And
> > > > > > because we are introducing this in table formats after
> > standardizing it
> > > > > > here, we can do that reliably.
> > > > > >
> > > > > > As I said in my last email, I think there's a difference between
> > > > > finiteness
> > > > > > and general "invariants" you might want. Vectors aren't vectors
> if
> > they
> > > > > > don't have a length and a direction. Other invariants don't work
> > the
> > > > same
> > > > > > way. A zero-length vector is still a valid vector and can be
> used,
> > just
> > > > > not
> > > > > > for certain distance metrics.
> > > > > >
> > > > > > I also want to clarify that when we looked into this more closely
> > and
> > > > > > recognized the mismatch between the two separate use cases, I
> > offered
> > > > to
> > > > > > split out the finite vector work so that we didn't cause any
> > headaches
> > > > > for
> > > > > > people who are interested in fixed-length list. It sounds like
> not
> > > > > > splitting out the use cases is causing a lot of headache. Vectors
> > for
> > > > our
> > > > > > use case need a finiteness guarantee at write time, and
> > fixed-length
> > > > > lists
> > > > > > need to store NaN and other values.
> > > > > >
> > > > > > It's okay to have two types and we should for two different use
> > cases.
> > > > > > Otherwise, we are forcing one to make an unnecessary compromise.
> I
> > > > don't
> > > > > > want to force everyone to exclude NaN and infinite values from
> > > > > fixed-length
> > > > > > lists. Seems reasonable that we should also be able to have
> vectors
> > > > that
> > > > > > exclude those values, since that is how they are used and we have
> > to
> > > > also
> > > > > > guarantee it for exchanging data with other existing databases.
> > > > > >
> > > > > > Ryan
> > > > > >
> > > > > > On Mon, Oct 19, 2026 at 3:12 AM Will Edwards via dev <
> > > > > > dev@parquet.apache.org>
> > > > > > wrote:
> > > > > >
> > > > > > > It reads well. Three small fixes:
> > > > > > >
> > > > > > > 1. **"back in May or so"** is ambiguous. The
> > > > > fixed-size-list-for-tensors
> > > > > > > framing is from Rok's opening post in May 2024. The May 2026
> > design
> > > > doc
> > > > > > was
> > > > > > > already mostly about performance and layout. "Originally"
> avoids
> > > > anyone
> > > > > > > correcting the date.
> > > > > > > 2. **"Spark's sparse feature vectors routinely use NaN"** isn't
> > quite
> > > > > > > right. The NaN-for-missing behaviour comes from
> > `VectorAssembler` and
> > > > > > > `Imputer` and applies to Spark ML vectors generally, dense or
> > sparse.
> > > > > > > "Spark's feature vectors, dense and sparse," fixes it.
> > > > > > > 3. There are typos: "Iirc" should be "IIRC", and "Spark,
> > BigQuery and
> > > > > > > DuckDB and pretty much…" has a doubled "and".
> > > > > > >
> > > > > > > The last sentence of the invariants paragraph could be clearer.
> > > > > "Although
> > > > > > > that doesn't mean readers shouldn't check on read" has a double
> > > > > negative
> > > > > > > and is easy to misread. "Though careful readers will still
> check
> > on
> > > > > read"
> > > > > > > is simpler.
> > > > > > >
> > > > > > > With those applied:
> > > > > > >
> > > > > > > ---
> > > > > > >
> > > > > > > Hi all,
> > > > > > >
> > > > > > > The scope of this discussion keeps moving :). IIRC it
> originally
> > was
> > > > > > about
> > > > > > > fixed-size lists for ML tensors, became a performance question,
> > and
> > > > is
> > > > > > now
> > > > > > > about whether vectors are mathematically valid. Each step made
> > sense,
> > > > > > but I
> > > > > > > don't think we have consensus on where we've ended up?
> > > > > > >
> > > > > > > I think there is much useful work in what we left behind. Many
> ML
> > > > > vectors
> > > > > > > aren't embeddings for search. E.g. Spark's feature vectors,
> > dense and
> > > > > > > sparse, routinely use NaN to mark missing values, and quantised
> > > > formats
> > > > > > > pack codes with scales in ways Parquet can't describe today.
> > > > > ML-oriented
> > > > > > > formats like Lance are moving towards tensors, quantisation and
> > > > > indexes.
> > > > > > A
> > > > > > > type that told engines how to interpret that kind of data would
> > > > enable
> > > > > > new
> > > > > > > things. An annotation on LIST<FLOAT> doesn't.
> > > > > > >
> > > > > > > My big concern with the current focus on embeddings needing a
> > logical
> > > > > > type
> > > > > > > is compatibility. Today LIST<FLOAT> is the common format for
> > > > > embeddings,
> > > > > > > and Spark, BigQuery, DuckDB and pretty much every other library
> > I can
> > > > > > think
> > > > > > > of offer distance functions on plain lists, so any engine can
> > use any
> > > > > > file.
> > > > > > > If engines start requiring a VECTOR annotation, existing files
> > and
> > > > > > anything
> > > > > > > rewritten by a tool that doesn't know the type stop working. If
> > they
> > > > > > don't
> > > > > > > require it, the annotation adds nothing. We've seen this before
> > with
> > > > > > > strings written without the UTF8 annotation, which engines
> still
> > have
> > > > > to
> > > > > > > cope with years later.
> > > > > > >
> > > > > > > On invariants, I agree some users may want the writer to check
> > what
> > > > > they
> > > > > > > write as they write it, but that can be a writer option rather
> > than a
> > > > > > type.
> > > > > > > Readers that want evidence could get it from extra statistics
> or
> > > > > > metadata,
> > > > > > > though careful readers will still check on read.
> > > > > > >
> > > > > > > If a finite fixed-length vector contract is useful to table
> > formats,
> > > > > > > perhaps Iceberg is the place for it, with Parquet storing plain
> > > > lists.
> > > > > > I'd
> > > > > > > raise the same worry there, though: if tables start rejecting
> > > > untagged
> > > > > > > columns, we've just moved the split up a layer.
> > > > > > >
> > > > > > > Best,
> > > > > > > Will
> > > > > > >
> > > > > > > On Sat, 17 Oct 2026 at 00:10, Ryan Blue <bl...@apache.org>
> wrote:
> > > > > > >
> > > > > > > > I agree with much of what Dan said, but I'll add some to this
> > > > thread
> > > > > as
> > > > > > > > well.
> > > > > > > >
> > > > > > > > First, I want to address the arguments that were pointed out:
> > > > > > > > 1. Other vector databases have finiteness requirements
> > > > > > > > 2. "Vectors are the hot stuff right now" -- let's jump on the
> > > > > bandwagon
> > > > > > > > 3. A general solution will take too long, finiteness will
> make
> > > > faster
> > > > > > > > progress
> > > > > > > > 4. To give a guarantee to readers that no INF/NaN/NULL will
> be
> > in
> > > > the
> > > > > > > > vector
> > > > > > > >
> > > > > > > > I don't think these represent our actual motivation or
> > position,
> > > > > which
> > > > > > > > makes them function as a straw-man. This is probably not what
> > was
> > > > > > > > intended, but I want to make sure that the misunderstanding
> > > > > > > > doesn't continue. For one thing, we aren't saying that we
> need
> > > > finite
> > > > > > > > vectors because they are "hot stuff".
> > > > > > > >
> > > > > > > > We want to build tools that use them on top of Parquet. I
> > think it
> > > > > > should
> > > > > > > > also be clear that we aren't advocating for finite elements
> > blindly
> > > > > > just
> > > > > > > > because other engines do (cargo cult design!). There are real
> > > > > > > > interoperability issues if we choose to go a different
> > direction so
> > > > > > this
> > > > > > > is
> > > > > > > > a "Chesterton's Fence" situation. It's important to
> understand
> > why
> > > > > > > > those databases
> > > > > > > > made that choice. I've also pointed this out as confirmation
> > that
> > > > > we're
> > > > > > > > going down a reasonable and established path.
> > > > > > > >
> > > > > > > > The third argument sounds like a simple misunderstanding.
> We're
> > > > > > pursuing
> > > > > > > a
> > > > > > > > logical type right now because that's forward compatible and
> we
> > > > want
> > > > > to
> > > > > > > > label the data properly and set expectations for the type.
> > Physical
> > > > > > type
> > > > > > > > changes break forward compatibility and need to be released
> in
> > a
> > > > new
> > > > > > > format
> > > > > > > > version . I think this is the source of the confusion since
> it
> > has
> > > > > been
> > > > > > > in
> > > > > > > > the same discussion, but the choice about finiteness is
> > separate.
> > > > > > > >
> > > > > > > > That brings us to the last argument, whether there is value
> in
> > > > > writers
> > > > > > > > participating in guarantees. I think there's value in adding
> > > > > > requirements
> > > > > > > > for writers. It's up to readers how they want to use those
> > > > > > requirements,
> > > > > > > > but a writer requirement allows a policy of "garbage in
> garbage
> > > > out"
> > > > > > > rather
> > > > > > > > than being forced to handle bad data. Why would a reader need
> > to
> > > > run
> > > > > > > > validation on every element just in case a writer had a bug?
> > And
> > > > why
> > > > > > > should
> > > > > > > > the potential for writer bugs mean that it has to be the
> > reader's
> > > > > > > problem?
> > > > > > > > If a writer produces bad data, we can absolutely reject it
> and
> > tell
> > > > > the
> > > > > > > > writer to fix the bug.
> > > > > > > >
> > > > > > > > I'm also puzzled by the assertion that writer guarantees have
> > > > value,
> > > > > > but
> > > > > > > > only if there are other, more specific guarantees in
> addition.
> > This
> > > > > > seems
> > > > > > > > to cede the point that a contract between readers and writers
> > makes
> > > > > > > sense,
> > > > > > > > but only if it is a _specific_ contract. But the proposed
> > additions
> > > > > > don't
> > > > > > > > make sense in the same way.
> > > > > > > >
> > > > > > > > Take, for example, the suggestion to also disallow
> zero-length
> > > > > vectors.
> > > > > > > At
> > > > > > > > a glance this seems similar, but I think it assumes a
> specific
> > > > > context
> > > > > > > and
> > > > > > > > is better suited to be a constraint for some use cases rather
> > than
> > > > > part
> > > > > > > of
> > > > > > > > what is considered valid for the type. That's because
> > zero-length
> > > > > > vectors
> > > > > > > > are valid vectors with a length. You can calculate Euclidean
> or
> > > > > > Manhattan
> > > > > > > > distance and use them in many contexts. They don't work if
> > you're
> > > > > > > > normalizing to a length of 1, but it's a choice to normalize
> > and
> > > > use
> > > > > a
> > > > > > > dot
> > > > > > > > product to calculate cosine distance. And at this point,
> we're
> > > > > talking
> > > > > > > > about the usefulness of restrictions for the type, not a
> > Platonic
> > > > > > ideal.
> > > > > > > >
> > > > > > > > I think the restriction that makes the most sense is that
> > vectors
> > > > > have
> > > > > > a
> > > > > > > > length and can be compared in distance calculations. 0-length
> > > > vectors
> > > > > > are
> > > > > > > > valid because they point somewhere and have a length. We
> > shouldn't
> > > > > > > restrict
> > > > > > > > uses to only cosine distance over normalized vectors. Those
> > are use
> > > > > > case
> > > > > > > > dependent choices and there are other valid ones. I also
> don't
> > > > think
> > > > > > that
> > > > > > > > the model used to produce a vector is part of the vector
> type,
> > just
> > > > > > like
> > > > > > > > the method used to produce a metric doesn't define its type:
> > cosine
> > > > > > > > similarity produces values between -1 and 1, but we store it
> > in a
> > > > > > > `float`,
> > > > > > > > not a `float(cosine)`.
> > > > > > > >
> > > > > > > > Why does it make sense to have a finiteness restriction as
> > part of
> > > > > the
> > > > > > > > type? Because that's what a (mathematical) vector is. A
> simple
> > > > search
> > > > > > > gives
> > > > > > > > a pretty common definition: "A vector is an object that has
> > both
> > > > > > > magnitude
> > > > > > > > and a direction". If a dimension/element is NaN, null, or
> > infinite,
> > > > > the
> > > > > > > > vector doesn't point anywhere and it doesn't have a
> magnitude.
> > Not
> > > > > only
> > > > > > > can
> > > > > > > > it not be used, it doesn't meet the most basic definition of
> > the
> > > > > type.
> > > > > > > >
> > > > > > > > Ryan
> > > > > > > >
> > > > > > > > On Thu, Oct 15, 2026 at 1:07 AM Jan Finis <jpfinis@gmail.com
> >
> > > > wrote:
> > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > Daniel,
> > > > > > > > >
> > > > > > > > > alright, so no physical optimizations intended.  We only
> want
> > > > this
> > > > > > for
> > > > > > > > > semantic reasons, not for any significant physical
> > optimization.
> > > > If
> > > > > > > we're
> > > > > > > > > all on the same page here, that is good 👍.
> > > > > > > > >
> > > > > > > > > To phrase my remaining problem with this more concretely:
> > > > > > > > > *Let's say we add this type. What cool new stuff can we now
> > do
> > > > that
> > > > > > we
> > > > > > > > > couldn't do before?*
> > > > > > > > > That is the litmus test that a type has to pass to be
> useful
> > > > IMHO.
> > > > > > And
> > > > > > > > here
> > > > > > > > > I can't come up with a tangible use case that wasn't
> possible
> > > > > before.
> > > > > > > > > Especially if we one day have fixed size lists in Parquet,
> > the
> > > > > > > remaining
> > > > > > > > > niche for this type becomes so small, that the added
> > complexity
> > > > > isn't
> > > > > > > > worth
> > > > > > > > > the benefit IMHO.
> > > > > > > > >
> > > > > > > > > As I guess most people on here are active maintainers of a
> > > > Parquet
> > > > > > > > > implementation: Concretely, what would you be using this
> > type for
> > > > > if
> > > > > > we
> > > > > > > > had
> > > > > > > > > it? Would you allow math only on fields tagged with this
> > type?
> > > > What
> > > > > > if
> > > > > > > > that
> > > > > > > > > math yields INF or NaN? Not allowed?
> > > > > > > > >
> > > > > > > > > Especially if we also discuss having both VECTOR and
> > > > FINITE_VECTOR.
> > > > > > How
> > > > > > > > > will you guys wire this up with the type systems of your
> > engines?
> > > > > Do
> > > > > > > your
> > > > > > > > > engines also have two types here? Do you type-decay a
> > > > FINITE_VECTOR
> > > > > > to
> > > > > > > a
> > > > > > > > > VECTOR once someone does a multiplication that could yield
> > INF?
> > > > By
> > > > > > > > itself,
> > > > > > > > > a FINITE_VECTOR sounds good. Holistically looked at from a
> > type
> > > > > > system
> > > > > > > > > perspective, it comes with a lot of complexity. Do the
> > benefits
> > > > > > really
> > > > > > > > > outweigh the complexity here? That's why I would rather
> have
> > the
> > > > > bare
> > > > > > > > > minimum: A fixed size array (list) of floats or at least
> > just a
> > > > > > vector
> > > > > > > > > without finite. Then all these detailed questions go away.
> > > > > > > > >
> > > > > > > > > I'll take this one step further... Why is there any debate
> > about
> > > > > this
> > > > > > > at
> > > > > > > > > > all? Why can't a type simply be added to the
> > *specification* as
> > > > > an
> > > > > > > > > > administrative detail and be done?
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Andrew,
> > > > > > > > >
> > > > > > > > > Parquet is supposed to be a lingua franca of data. The
> > > > > specification
> > > > > > is
> > > > > > > > the
> > > > > > > > > authoritative truth for this language and all
> > implementations are
> > > > > > > > expected
> > > > > > > > > to implement as much as possible from this spec, so
> anything
> > in
> > > > the
> > > > > > > spec
> > > > > > > > > does causes implementation effort. So just adding anything
> to
> > > > this
> > > > > > spec
> > > > > > > > > without debate is not a good idea. It is even the other way
> > > > around:
> > > > > > We
> > > > > > > > need
> > > > > > > > > to have way higher rigor with the spec than we need to have
> > with
> > > > > any
> > > > > > > > > implementations. Implementations can be changed, the spec
> > needs
> > > > to
> > > > > be
> > > > > > > > > stable.
> > > > > > > > >
> > > > > > > > > A tag-only type that doesn't come with any restrictions can
> > be
> > > > > added
> > > > > > > with
> > > > > > > > > little effort. But here we want to tie restrictions to it
> (no
> > > > > > NaN/Inf),
> > > > > > > > so
> > > > > > > > > we would expect implementations to enforce these. So there
> is
> > > > some
> > > > > > > > > implementation effort expected here.
> > > > > > > > >
> > > > > > > > > Also, even if we only wanted a tag type, I would still
> argue
> > that
> > > > > you
> > > > > > > > > shouldn't add those without a debate. What if A wants XML,
> B
> > > > wants
> > > > > > > YAML,
> > > > > > > > C
> > > > > > > > > wants quaternions, D wants address, E wants phone number.
> If
> > we
> > > > > just
> > > > > > > > added
> > > > > > > > > any types that are "just tags", Parquet logical types would
> > just
> > > > > > > become a
> > > > > > > > > trash dump of any type that anyone ever thought to be
> > useful. So
> > > > > > > rigorous
> > > > > > > > > assessment of each new type candidate is necessary.
> > > > > > > > >
> > > > > > > > > Finally, many engines couple parquet with another type
> system
> > > > > (e.g.,
> > > > > > > > Arrow
> > > > > > > > > or SQL). So even with a tag only type, you at least still
> > have
> > > > the
> > > > > > > > > implementation burden of wiring this up with your engine's
> > type
> > > > > > system.
> > > > > > > > >
> > > > > > > > > So, just because logical types are "just" logical, that
> > doesn't
> > > > > mean
> > > > > > we
> > > > > > > > can
> > > > > > > > > add them without any rigor.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > A few naive questions:
> > > > > > > > > > 1. Would a physical definition with NANs & Inf+/- allow
> > > > efficient
> > > > > > > > > > encodings like ALP?
> > > > > > > > > > 2. If the physical representation allows values that make
> > > > > encodings
> > > > > > > > like
> > > > > > > > > > ALP impossible, can a tighter Logical type allow
> different
> > > > > > encodings?
> > > > > > > > > > 3. Can we RLE or binpack a vector?
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > The simple general answer here is that all these things do
> > not
> > > > > > require
> > > > > > > a
> > > > > > > > > logical type. Writers can already do any such optimization
> > if the
> > > > > > > actual
> > > > > > > > > data contains no NaNs & Infs. All these physical
> > optimizations
> > > > just
> > > > > > > > require
> > > > > > > > > no actual NaNs/Infs to be present in a page, they don't
> need
> > a
> > > > > > > guarantee
> > > > > > > > > that the whole file won't contain any.
> > > > > > > > >
> > > > > > > > > Cheers,
> > > > > > > > > Jan
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Am Do., 15. Okt. 2026 um 04:28 Uhr schrieb <
> > > > > kurtiscwright@gmail.com
> > > > > > >:
> > > > > > > > >
> > > > > > > > > > Hi Everyone,
> > > > > > > > > >
> > > > > > > > > > Glad to see a Vector type being added. I have a few use
> > cases I
> > > > > > want
> > > > > > > to
> > > > > > > > > > pursue once we have an agreed upon definition.
> > > > > > > > > >
> > > > > > > > > > In regards to that definition discussion. I feel
> > comfortable
> > > > with
> > > > > > the
> > > > > > > > > idea
> > > > > > > > > > of the semantic meaning Dan raises though I can’t shake a
> > > > slight
> > > > > > > > > discomfort
> > > > > > > > > > (maybe FOMO) with limiting values in the physical
> > definition.
> > > > > > > > > >
> > > > > > > > > > A few naive questions:
> > > > > > > > > > 1. Would a physical definition with NANs & Inf+/- allow
> > > > efficient
> > > > > > > > > > encodings like ALP?
> > > > > > > > > > 2. If the physical representation allows values that make
> > > > > encodings
> > > > > > > > like
> > > > > > > > > > ALP impossible, can a tighter Logical type allow
> different
> > > > > > encodings?
> > > > > > > > > > 3. Can we RLE or binpack a vector?
> > > > > > > > > >
> > > > > > > > > > Best Regards,
> > > > > > > > > > Kurtis C. Wright
> > > > > > > > > > > On Oct 14, 2026, at 12:45, Daniel Weeks <
> > dweeks@apache.org>
> > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > Hey Jan,
> > > > > > > > > > >
> > > > > > > > > > > There's a lot that I disagree with respect to the
> > framing of
> > > > > > these
> > > > > > > > > > > arguments and how we should be thinking about
> introducing
> > > > > vector
> > > > > > > as a
> > > > > > > > > > > logical type.  There's a repeated tendency to let the
> > > > physical
> > > > > > > > > > > representation or performance drive discussion around
> > types
> > > > as
> > > > > > > > opposed
> > > > > > > > > to
> > > > > > > > > > > what it represents.
> > > > > > > > > > >
> > > > > > > > > > > In parquet, establishing semantic meaning is the
> purpose
> > of
> > > > > > logical
> > > > > > > > > > types.
> > > > > > > > > > > The logical type separates how to interpret a value
> from
> > its
> > > > > > > physical
> > > > > > > > > > > representation.  We can represent a timestamp with
> INT64
> > or
> > > > > even
> > > > > > a
> > > > > > > > > string
> > > > > > > > > > > value, so why have a type for it? Even though a
> > > > > FIXED_SIZED_LIST
> > > > > > > and
> > > > > > > > > > VECTOR
> > > > > > > > > > > may have the exact same representation and performance,
> > they
> > > > > have
> > > > > > > > > > different
> > > > > > > > > > > semantic meaning.  Databases and engines across the
> > ecosystem
> > > > > > > already
> > > > > > > > > > > recognize this differentiation and have separate
> > > > > representations.
> > > > > > > > The
> > > > > > > > > > > choice to separate these isn't because the discussion
> is
> > too
> > > > > > > tedious;
> > > > > > > > > > > they're fundamentally different things.
> > > > > > > > > > >
> > > > > > > > > > > The finite requirement is what makes a vector a
> > reflection of
> > > > > > what
> > > > > > > it
> > > > > > > > > > > represents.  As we've discussed in the thread,
> non-finite
> > > > > values
> > > > > > in
> > > > > > > > the
> > > > > > > > > > > representation violate the properties of a vector and
> the
> > > > value
> > > > > > > > ceases
> > > > > > > > > to
> > > > > > > > > > > be a vector.  On the other hand, a 0 vector is still a
> > vector
> > > > > > (you
> > > > > > > > can
> > > > > > > > > > > argue the utility of it, but it is a valid vector).
> > > > > > > Implementations
> > > > > > > > > can
> > > > > > > > > > > choose what they validate or how to handle execution
> > errors;
> > > > > > that's
> > > > > > > > an
> > > > > > > > > > > implementation concern, not a requirement.  Operations
> > > > > performed
> > > > > > > on a
> > > > > > > > > > > vector depend on the context in which they are used,
> but
> > the
> > > > > > > > > > representation
> > > > > > > > > > > doesn't need to solve scenarios where that application
> > > > doesn't
> > > > > > make
> > > > > > > > > > sense.
> > > > > > > > > > >
> > > > > > > > > > > The guarantees or context needed to use a vector do not
> > need
> > > > to
> > > > > > be
> > > > > > > > > > inherent
> > > > > > > > > > > in the type for it to be a useful type.  We don't need
> > the
> > > > > > currency
> > > > > > > > > used
> > > > > > > > > > > for decimal or whether a floating point is a normalized
> > value
> > > > > as
> > > > > > > part
> > > > > > > > > of
> > > > > > > > > > > the type to make use of those types.  This conflates
> the
> > use
> > > > > > cases
> > > > > > > > with
> > > > > > > > > > the
> > > > > > > > > > > type system and unnecessarily attempts to draw the
> > context
> > > > into
> > > > > > the
> > > > > > > > > type
> > > > > > > > > > > definition.  Similarly, you don't need to capture
> > > > > normalization,
> > > > > > > > model,
> > > > > > > > > > > etc. as part of the type for it to be useful.
> > > > > > > > > > >
> > > > > > > > > > > So the reason we want to add a vector type (or
> > FINITE_VECTOR)
> > > > > is
> > > > > > > > > because
> > > > > > > > > > we
> > > > > > > > > > > want to store vectors with the semantic meaning and
> > > > > requirements
> > > > > > > > based
> > > > > > > > > on
> > > > > > > > > > > what they represent.
> > > > > > > > > > >
> > > > > > > > > > > -Dan
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >> On Wed, Oct 14, 2026 at 1:22 AM Jan Finis <
> > > > jpfinis@gmail.com>
> > > > > > > > wrote:
> > > > > > > > > > >>
> > > > > > > > > > >> A bit late to the party, but here we go:
> > > > > > > > > > >>
> > > > > > > > > > >> I have the strong sentiment that the FINITE_VECTOR
> type
> > as
> > > > it
> > > > > is
> > > > > > > > > > currently
> > > > > > > > > > >> being discussed will not be very useful and just
> > increase
> > > > > > > > > implementation
> > > > > > > > > > >> complexity. The arguments for the type are all quite
> > weak
> > > > > > (details
> > > > > > > > see
> > > > > > > > > > >> below). The FINITE part doesn't seem to yield any
> value
> > and
> > > > is
> > > > > > > not a
> > > > > > > > > > useful
> > > > > > > > > > >> restriction by itself. The vector (i.e. fixed length)
> > part
> > > > > > should
> > > > > > > > > > rather be
> > > > > > > > > > >> generalized instead of making a one-off thing. In
> > general,
> > > > the
> > > > > > > > > existing
> > > > > > > > > > >> types are already working well for uncompressed
> > vectors. The
> > > > > > real
> > > > > > > > > > challenge
> > > > > > > > > > >> that would bring real benefits is supporting
> > quantization.
> > > > > This
> > > > > > is
> > > > > > > > > where
> > > > > > > > > > >> you get orders of magnitude improvements to storage
> > space
> > > > and
> > > > > > > > > processing
> > > > > > > > > > >> speed.
> > > > > > > > > > >>
> > > > > > > > > > >> Let's first discuss the FINITE part. So far, the only
> > > > > arguments
> > > > > > I
> > > > > > > > have
> > > > > > > > > > read
> > > > > > > > > > >> for this are:
> > > > > > > > > > >>
> > > > > > > > > > >> 1. many other vector databases have it
> > > > > > > > > > >> 2. vectors are the hot stuff right now, so Parquet
> > better
> > > > > boast
> > > > > > > > having
> > > > > > > > > > >> "native" type support for vectors
> > > > > > > > > > >> 3. by focussing on this common special case, we can
> make
> > > > rapid
> > > > > > > > > progress,
> > > > > > > > > > >> while a general solution would take too long to
> specify
> > > > > > > > > > >> 4. to give a guarantee to readers that no INF/NaN/NULL
> > will
> > > > be
> > > > > > in
> > > > > > > > the
> > > > > > > > > > >> vector
> > > > > > > > > > >>
> > > > > > > > > > >> the first two arguments are quite weak by themselves.
> > Vector
> > > > > > > > databases
> > > > > > > > > > are
> > > > > > > > > > >> very specialized engines. Such engines can of course
> be
> > very
> > > > > > > > > > opinionated on
> > > > > > > > > > >> their data types. A general format like Parquet must
> > instead
> > > > > > weigh
> > > > > > > > > > whether
> > > > > > > > > > >> a special type is worth the added complexity over a
> more
> > > > > > general,
> > > > > > > > > widely
> > > > > > > > > > >> applicable type. In this case, it is IMHO not at all.
> > And
> > > > just
> > > > > > > > having
> > > > > > > > > a
> > > > > > > > > > >> type called "vector" for advertisement reasons to make
> > > > Parquet
> > > > > > > > appear
> > > > > > > > > > more
> > > > > > > > > > >> modern seems a non-goal in the first place.
> > > > > > > > > > >>
> > > > > > > > > > >> The third "time to market" argument is valid for
> systems
> > > > that
> > > > > > > don't
> > > > > > > > > > need a
> > > > > > > > > > >> stable interface. It is a very dangerous attitude for
> > > > formats
> > > > > > like
> > > > > > > > > > Parquet
> > > > > > > > > > >> that must be stable long term and should have a
> > > > consolidated,
> > > > > > > > concise,
> > > > > > > > > > >> non-redundant type system. For such long term storage
> > > > formats,
> > > > > > we
> > > > > > > > need
> > > > > > > > > > to
> > > > > > > > > > >> take the time to build the right thing, as any
> > shortcuts we
> > > > > take
> > > > > > > > will
> > > > > > > > > > stay
> > > > > > > > > > >> as blemishes in the format forever. If I read the
> > discussion
> > > > > > > > > correctly,
> > > > > > > > > > >> this is actually the main argument as of now: We have
> > found
> > > > > that
> > > > > > > it
> > > > > > > > is
> > > > > > > > > > hard
> > > > > > > > > > >> to align on how a fixed size list type should look
> > like, so
> > > > we
> > > > > > > want
> > > > > > > > to
> > > > > > > > > > go
> > > > > > > > > > >> the easier route of a specialized type. This seems
> like
> > a
> > > > > > > > > > >> dangerous shortcut fueled by feature recency bias. Ask
> > > > > yourself:
> > > > > > > > Would
> > > > > > > > > > we
> > > > > > > > > > >> want this, if Parquet already had a fixed size list
> > type?
> > > > > > > > > > >>
> > > > > > > > > > >> So finally, the only technical argument is the fourth
> > one:
> > > > "It
> > > > > > > > signals
> > > > > > > > > > >> semantics and ensures that no bad data lands in the
> > > > format". I
> > > > > > see
> > > > > > > > two
> > > > > > > > > > big
> > > > > > > > > > >> problems with this:
> > > > > > > > > > >>
> > > > > > > > > > >> 1. Readers *still* need to test for NaN / INF because
> > > > Parquet
> > > > > > > files
> > > > > > > > > may
> > > > > > > > > > be
> > > > > > > > > > >> corrupt. Nothing can stop a malicious or buggy writer
> > from
> > > > > > > > physically
> > > > > > > > > > >> writing NaN or INF. *So you save no single line of
> code
> > in a
> > > > > > > reader.
> > > > > > > > > You
> > > > > > > > > > >> save no branches, as you still need to handle the
> > INF/NaN
> > > > > case.*
> > > > > > > So
> > > > > > > > > this
> > > > > > > > > > >> doesn't help with performance at all. The only
> > difference
> > > > is,
> > > > > > that
> > > > > > > > the
> > > > > > > > > > >> error message changes from "found an invalid INF value
> > in
> > > > the
> > > > > > > data"
> > > > > > > > to
> > > > > > > > > > >> "Your parquet file seems to be corrupt". It is
> actually
> > > > quite
> > > > > > the
> > > > > > > > > > opposite,
> > > > > > > > > > >> readers and writers get more complex having to handle
> > this
> > > > new
> > > > > > > type.
> > > > > > > > > > >>
> > > > > > > > > > >> If you now argue that it's not about performance, it's
> > about
> > > > > > > defined
> > > > > > > > > > >> semantics and erroring out early in writers, then the
> > second
> > > > > > > problem
> > > > > > > > > > >> arises:
> > > > > > > > > > >>
> > > > > > > > > > >> 2. The proposed no-NaN/INF/NULL guarantee is just one
> of
> > > > > > multiple
> > > > > > > > > > >> guarantees that you need for sensible vector
> > processing. If
> > > > > "no
> > > > > > > > > > >> INF/NaN/NULL" was the only guarantee you needed, I
> would
> > > > agree
> > > > > > > that
> > > > > > > > > > >> FINITE_VECTOR is useful, but it is by no means the
> only
> > > > > > guarantee
> > > > > > > > you
> > > > > > > > > > need,
> > > > > > > > > > >> instead, depending on the distance metric, you also
> > need the
> > > > > > > > following
> > > > > > > > > > >> guarantees. E.g.:
> > > > > > > > > > >>
> > > > > > > > > > >>   - No vector may be the 0 vector, otherwise, e.g.,
> > cosine
> > > > > > > > similarity
> > > > > > > > > is
> > > > > > > > > > >>   undefined. Cosine similarity is often used as
> > default, so
> > > > > this
> > > > > > > is
> > > > > > > > > > >> actually
> > > > > > > > > > >>   quite important in practice.
> > > > > > > > > > >>   - If we want to use a dot product, then all vectors
> > need
> > > > to
> > > > > > > > > guarantee
> > > > > > > > > > a
> > > > > > > > > > >>   length of 1. This would actually be quite useful for
> > > > > > performance
> > > > > > > > > > >> reasons.
> > > > > > > > > > >>   - The vectors must come from the same model as the
> > vector
> > > > > used
> > > > > > > for
> > > > > > > > > the
> > > > > > > > > > >>   search. Otherwise, the search yields a numeric
> > result, but
> > > > > > that
> > > > > > > > > result
> > > > > > > > > > >> has
> > > > > > > > > > >>   no sensible meaning.
> > > > > > > > > > >>
> > > > > > > > > > >> So, what use is a type that just guarantees part of
> the
> > > > things
> > > > > > you
> > > > > > > > > need
> > > > > > > > > > for
> > > > > > > > > > >> vector search but not others? You still need
> additional
> > > > > > guarantees
> > > > > > > > in
> > > > > > > > > > >> addition to that type. What makes INF/NaN so important
> > in
> > > > > > > comparison
> > > > > > > > > to,
> > > > > > > > > > >> say, "no 0 vector present", that it warrants an own
> > type,
> > > > > while
> > > > > > > the
> > > > > > > > > > other
> > > > > > > > > > >> guarantees do not? If we really want this type as a
> > basis
> > > > for
> > > > > > > > > > cross-engine
> > > > > > > > > > >> sharing of vectors, then it needs to give *all*
> > guarantees.
> > > > > > > > > > >>
> > > > > > > > > > >> I'm not against a specialized vector type if it really
> > > > carries
> > > > > > > *all*
> > > > > > > > > > >> information to be useful by itself. If it carried all
> > that
> > > > > > > > > information,
> > > > > > > > > > >> then a reading engine could indeed validate that only
> > > > > meaningful
> > > > > > > > > vector
> > > > > > > > > > >> searches are executed. But as long as it is just a
> > > > half-baked
> > > > > > > thing,
> > > > > > > > > it
> > > > > > > > > > is
> > > > > > > > > > >> not useful by itself and just a burden on each
> > implementer
> > > > and
> > > > > > > > > > maintainer.
> > > > > > > > > > >> If we need extra information anyway, why not just
> stick
> > to a
> > > > > > > general
> > > > > > > > > > type
> > > > > > > > > > >> like VECTOR (without FINITE).
> > > > > > > > > > >>
> > > > > > > > > > >> If you're a proponent of FINITE_VECTOR, ask yourself
> (or
> > > > tell
> > > > > > us)
> > > > > > > > what
> > > > > > > > > > >> advantage you will *really* get over just a
> (potentially
> > > > fixed
> > > > > > > size)
> > > > > > > > > > List
> > > > > > > > > > >> of Float? As discussed, it's not performance. It's
> also
> > not
> > > > a
> > > > > > > > semantic
> > > > > > > > > > >> marker that is sufficient by itself. It's just a small
> > hint
> > > > > that
> > > > > > > > > allows
> > > > > > > > > > >> writers to error out once they see a NaN or INF in the
> > > > input,
> > > > > > > while
> > > > > > > > > they
> > > > > > > > > > >> will still happily write a bogus 0 vector. This seems
> > to be
> > > > of
> > > > > > > zero
> > > > > > > > > > >> benefit.
> > > > > > > > > > >>
> > > > > > > > > > >> Anecdotally, we currently handle petabytes of vectors
> in
> > > > > > > production
> > > > > > > > in
> > > > > > > > > > >> Data360. We simply store them in a List of Float, with
> > an
> > > > > > > optimized
> > > > > > > > > > R-Level
> > > > > > > > > > >> reader implementation that rapidly checks that all
> > vectors
> > > > are
> > > > > > the
> > > > > > > > > same
> > > > > > > > > > >> length. This solution works just fine. The overhead of
> > > > R-level
> > > > > > > > > decoding
> > > > > > > > > > is
> > > > > > > > > > >> barely measurable. The check is super fast and
> R-levels
> > are
> > > > > tiny
> > > > > > > in
> > > > > > > > > > >> comparison to the payload vectors, so also the added
> > I/O is
> > > > > > > > > negligible.
> > > > > > > > > > >> This is also why I'm so firm in my sentiment: We are
> > > > running a
> > > > > > > > > > production
> > > > > > > > > > >> grade system processing trillions of vectors right now
> > and
> > > > see
> > > > > > no
> > > > > > > > big
> > > > > > > > > > >> issues with the types that Parquet currently provides.
> > > > > > > > > > >>
> > > > > > > > > > >> As I don't just want to be a naysayer and point out
> > problems
> > > > > > with
> > > > > > > > the
> > > > > > > > > > >> current design, here are the directions that I would
> > propose
> > > > > > > indeed:
> > > > > > > > > > >>
> > > > > > > > > > >>   - a general fixed size list type. Don't do shortcuts
> > just
> > > > > > > because
> > > > > > > > > the
> > > > > > > > > > >>   discussion is too tedious.
> > > > > > > > > > >>   - a vector type that indeed carries enough
> > information to
> > > > be
> > > > > > > > really
> > > > > > > > > > >>   useful for vector search by itself. (e.g., model
> name
> > /
> > > > > > version,
> > > > > > > > > > >> optional
> > > > > > > > > > >>   guarantees for no 0 vector / normalized vector
> > length).
> > > > With
> > > > > > > such
> > > > > > > > a
> > > > > > > > > > >> type,
> > > > > > > > > > >>   you could really do well defined vector searches
> even
> > if
> > > > the
> > > > > > > > writing
> > > > > > > > > > >> engine
> > > > > > > > > > >>   is completely different to the reading one.
> > > > > > > > > > >>   - Standardize quantizations. This will be a real
> game
> > > > > changer
> > > > > > > for
> > > > > > > > > > >>   performance.
> > > > > > > > > > >>
> > > > > > > > > > >> Cheers,
> > > > > > > > > > >> Jan
> > > > > > > > > > >>
> > > > > > > > > > >>
> > > > > > > > > > >>
> > > > > > > > > > >>
> > > > > > > > > > >>
> > > > > > > > > > >>
> > > > > > > > > > >>> Am Mi., 14. Okt. 2026 um 03:36 Uhr schrieb Gang Wu <
> > > > > > > > ustcwg@gmail.com
> > > > > > > > > >:
> > > > > > > > > > >>>
> > > > > > > > > > >>> I think separating them into FINITE_VECTOR and
> > > > > > FIXED_LENGTH_LIST
> > > > > > > > is a
> > > > > > > > > > >> good
> > > > > > > > > > >>> idea, though I think renaming FINITE_VECTOR to VECTOR
> > looks
> > > > > > more
> > > > > > > > > > natural.
> > > > > > > > > > >>> Should we move forward with proposing
> > FINITE_VECTOR/VECTOR
> > > > > > first?
> > > > > > > > > > >>>
> > > > > > > > > > >>> On Tue, Oct 13, 2026 at 6:26 PM Alkis Evlogimenos via
> > dev <
> > > > > > > > > > >>> dev@parquet.apache.org> wrote:
> > > > > > > > > > >>>
> > > > > > > > > > >>>> I suggest we go with VECTOR for the finite vector to
> > keep
> > > > > the
> > > > > > > name
> > > > > > > > > the
> > > > > > > > > > >>> same
> > > > > > > > > > >>>> as other engines/type systems. If we want a
> > non-finite one
> > > > > in
> > > > > > > the
> > > > > > > > > > >> future
> > > > > > > > > > >>> we
> > > > > > > > > > >>>> can name that something else or we add fixed size
> list
> > > > that
> > > > > > will
> > > > > > > > > serve
> > > > > > > > > > >>> the
> > > > > > > > > > >>>> same purpose without a special name.
> > > > > > > > > > >>>>
> > > > > > > > > > >>>>
> > > > > > > > > > >>>> On Thu, Oct 8, 2026 at 10:20 PM Ryan Blue <
> > > > blue@apache.org>
> > > > > > > > wrote:
> > > > > > > > > > >>>>
> > > > > > > > > > >>>>> A logical type would:
> > > > > > > > > > >>>>> - Describe data for users and readers to understand
> > what
> > > > > the
> > > > > > > file
> > > > > > > > > > >>>> contains
> > > > > > > > > > >>>>> - Signal that repetition and definition levels will
> > > > > reflect a
> > > > > > > > fixed
> > > > > > > > > > >>>> number
> > > > > > > > > > >>>>> of elements, which implementations can use to
> > optimize
> > > > > > decoding
> > > > > > > > > > >>>>> - Guarantee that there are no null elements
> (because
> > the
> > > > > type
> > > > > > > > > cannot
> > > > > > > > > > >> be
> > > > > > > > > > >>>>> applied to a list with optional elements)
> > > > > > > > > > >>>>> - [FINITE_VECTOR] Guarantee that no elements are
> NaN
> > or
> > > > > > > Infinity
> > > > > > > > > > >> values
> > > > > > > > > > >>>> so
> > > > > > > > > > >>>>> that the vectors have a finite length and are
> > suitable
> > > > for
> > > > > > > > distance
> > > > > > > > > > >>>>> calculations
> > > > > > > > > > >>>>>
> > > > > > > > > > >>>>> The finite value restriction is a restriction on
> > writers
> > > > to
> > > > > > > > prevent
> > > > > > > > > > >> bad
> > > > > > > > > > >>>>> data from being written.
> > > > > > > > > > >>>>>
> > > > > > > > > > >>>>> Ignoring logical types that are not understood is a
> > rule
> > > > > for
> > > > > > > > > > >> readers. A
> > > > > > > > > > >>>>> reader can handle vectors and pass it on to another
> > > > layer.
> > > > > > For
> > > > > > > > > > >>> instance,
> > > > > > > > > > >>>> a
> > > > > > > > > > >>>>> Parquet reader that doesn't support geometry passes
> > > > values
> > > > > as
> > > > > > > > > > >> `binary`.
> > > > > > > > > > >>>>> That binary value can be passed to a geometry UDF,
> > even
> > > > if
> > > > > > the
> > > > > > > > > > >>> underlying
> > > > > > > > > > >>>>> engine and Parquet reader doesn't have support.
> > > > > > > > > > >>>>>
> > > > > > > > > > >>>>> A tool that doesn't understand FINITE_VECTOR can
> > > > similarly
> > > > > > > pass a
> > > > > > > > > > >> list
> > > > > > > > > > >>> of
> > > > > > > > > > >>>>> floats along to a UDF. But it is still important to
> > the
> > > > > user
> > > > > > > that
> > > > > > > > > bad
> > > > > > > > > > >>>>> values are not present in the data. In your
> example,
> > bad
> > > > > > values
> > > > > > > > are
> > > > > > > > > > >>>> vectors
> > > > > > > > > > >>>>> that cause the centroid calculation to fail, like
> > vectors
> > > > > > with
> > > > > > > > > > >> infinite
> > > > > > > > > > >>>>> length.
> > > > > > > > > > >>>>>
> > > > > > > > > > >>>>> Ryan
> > > > > > > > > > >>>>>
> > > > > > > > > > >>>>> On Thu, Oct 8, 2026 at 12:54 PM Will Edwards via
> dev
> > <
> > > > > > > > > > >>>>> dev@parquet.apache.org>
> > > > > > > > > > >>>>> wrote:
> > > > > > > > > > >>>>>
> > > > > > > > > > >>>>>> Could we clarify the use cases for a VECTOR
> > annotation
> > > > on
> > > > > > > > > > >>> LIST<FLOAT>?
> > > > > > > > > > >>>>>> Specifically, what does it make possible that's
> > > > impossible
> > > > > > > > today,
> > > > > > > > > > >> and
> > > > > > > > > > >>>>> what
> > > > > > > > > > >>>>>> does it make easier that's currently hard?
> > > > > > > > > > >>>>>>
> > > > > > > > > > >>>>>> I'm struggling to find a strong answer. I'd like
> > every
> > > > > > engine
> > > > > > > to
> > > > > > > > > > >>> offer
> > > > > > > > > > >>>>>> vector functions on lists of floats without
> needing
> > a
> > > > new
> > > > > > > > logical
> > > > > > > > > > >>> type.
> > > > > > > > > > >>>>>>
> > > > > > > > > > >>>>>> To be clear, I'm still enthusiastic about a vector
> > type
> > > > > that
> > > > > > > > > > >>> describes
> > > > > > > > > > >>>>> how
> > > > > > > > > > >>>>>> to unpack and scale composite representations
> > (quantised
> > > > > > codes
> > > > > > > > > plus
> > > > > > > > > > >>>>> scales,
> > > > > > > > > > >>>>>> codebooks, etc.), in the direction ML is going.
> That
> > > > adds
> > > > > > > > > > >> information
> > > > > > > > > > >>>>>> engines don't have. A label on LIST<FLOAT>
> doesn't:
> > > > > engines
> > > > > > > > > already
> > > > > > > > > > >>>>>> understand the physical type, and many already
> offer
> > > > > vector
> > > > > > > > > > >> functions
> > > > > > > > > > >>>> on
> > > > > > > > > > >>>>>> it. I don't see a performance benefit, and
> > NaN/null/Inf
> > > > > > checks
> > > > > > > > are
> > > > > > > > > > >>>> cheap
> > > > > > > > > > >>>>>> enough that engines should do them anyway. The
> label
> > > > also
> > > > > > > > doesn't
> > > > > > > > > > >> say
> > > > > > > > > > >>>>> what
> > > > > > > > > > >>>>>> you can do with the column. Knowing that a column
> > holds
> > > > > > > > embeddings
> > > > > > > > > > >>>>> doesn't
> > > > > > > > > > >>>>>> say how they can be used, just as zipcode and age
> > are
> > > > both
> > > > > > > > > integers
> > > > > > > > > > >>> but
> > > > > > > > > > >>>>>> multiplying them is meaningless.
> > > > > > > > > > >>>>>>
> > > > > > > > > > >>>>>> My concern is the opposite effect. We're
> > establishing
> > > > that
> > > > > > > > engines
> > > > > > > > > > >>>> should
> > > > > > > > > > >>>>>> ignore logical types they don't understand. So a
> > tool
> > > > that
> > > > > > > > doesn't
> > > > > > > > > > >>> know
> > > > > > > > > > >>>>>> VECTOR will read the column as LIST<FLOAT>, let
> the
> > user
> > > > > > > compute
> > > > > > > > > > >>>>> centroids
> > > > > > > > > > >>>>>> (say), and write plain LIST<FLOAT> back out. If
> > other
> > > > > > engines
> > > > > > > > > > >>> restrict
> > > > > > > > > > >>>>>> embedding functions to VECTOR-annotated columns,
> > that
> > > > file
> > > > > > > stops
> > > > > > > > > > >>>> working
> > > > > > > > > > >>>>>> with them, through no fault of the user. An
> > annotation
> > > > > that
> > > > > > > > > doesn't
> > > > > > > > > > >>>>> change
> > > > > > > > > > >>>>>> what the data is shouldn't be able to make it
> > unusable.
> > > > > > > > > > >>>>>>
> > > > > > > > > > >>>>>> Am I missing a use case?
> > > > > > > > > > >>>>>>
> > > > > > > > > > >>>>>> Best, Will
> > > > > > > > > > >>>>>>
> > > > > > > > > > >>>>>> On Thu, 8 Oct 2026 at 19:53, Steven Wu <
> > > > > > stevenz3wu@gmail.com>
> > > > > > > > > > >> wrote:
> > > > > > > > > > >>>>>>
> > > > > > > > > > >>>>>>> I also think a FINITE_VECTOR type brings a lot of
> > > > values
> > > > > to
> > > > > > > > many
> > > > > > > > > > >>> use
> > > > > > > > > > >>>>>> cases.
> > > > > > > > > > >>>>>>> It is would be great to make progress on the
> > > > > FINITE_VECTOR,
> > > > > > > and
> > > > > > > > > > >> not
> > > > > > > > > > >>>> to
> > > > > > > > > > >>>>>> fit
> > > > > > > > > > >>>>>>> every use cases into the FIXED_LENGTH_LIST.
> > > > > > > > > > >>>>>>>
> > > > > > > > > > >>>>>>> On Mon, Oct 5, 2026 at 1:26 PM Ryan Blue <
> > > > > blue@apache.org>
> > > > > > > > > > >> wrote:
> > > > > > > > > > >>>>>>>
> > > > > > > > > > >>>>>>>> Going through this thread again, it looks like
> > there
> > > > > are a
> > > > > > > few
> > > > > > > > > > >>>> things
> > > > > > > > > > >>>>>>> that
> > > > > > > > > > >>>>>>>> people generally agree on:
> > > > > > > > > > >>>>>>>>
> > > > > > > > > > >>>>>>>> - In the short term, we want to introduce a
> > logical
> > > > type
> > > > > > or
> > > > > > > > > > >>> types;
> > > > > > > > > > >>>> a
> > > > > > > > > > >>>>>>> better
> > > > > > > > > > >>>>>>>> physical representation can be done later. We
> can
> > > > focus
> > > > > on
> > > > > > > > > > >>> logical
> > > > > > > > > > >>>>>> types.
> > > > > > > > > > >>>>>>>> - There is precedent in logical types for
> > restricting
> > > > > > > values,
> > > > > > > > > > >>> like
> > > > > > > > > > >>>>> UTF8
> > > > > > > > > > >>>>>>>> strings to contain only UTF-8 codepoints. I
> think
> > we
> > > > > also
> > > > > > > > agree
> > > > > > > > > > >>>> that
> > > > > > > > > > >>>>> we
> > > > > > > > > > >>>>>>>> could restrict vector elements to non-nested or
> > > > > primitive
> > > > > > > > > > >> types.
> > > > > > > > > > >>>>>>>> - There are many examples of databases that
> > restrict
> > > > > their
> > > > > > > > > > >> vector
> > > > > > > > > > >>>>> types
> > > > > > > > > > >>>>>>> to
> > > > > > > > > > >>>>>>>> non-null, finite elements.
> > > > > > > > > > >>>>>>>> - Some people want to have write-time guarantees
> > that
> > > > > > > vectors
> > > > > > > > > > >>> have
> > > > > > > > > > >>>>> only
> > > > > > > > > > >>>>>>>> non-null, finite elements
> > > > > > > > > > >>>>>>>> - Some people want to extend this work to
> include
> > > > > tensors,
> > > > > > > > > > >>> shorter
> > > > > > > > > > >>>>>>> floating
> > > > > > > > > > >>>>>>>> point representations, variable lengths, NaN or
> > > > Infinity
> > > > > > > > > > >>> elements,
> > > > > > > > > > >>>>> etc.
> > > > > > > > > > >>>>>>>>
> > > > > > > > > > >>>>>>>> It sounds like we are nearly there, but have two
> > > > halves
> > > > > of
> > > > > > > the
> > > > > > > > > > >>>>>> community
> > > > > > > > > > >>>>>>>> that care about different things. That just
> means
> > we
> > > > > > should
> > > > > > > > > > >> have
> > > > > > > > > > >>>>>> multiple
> > > > > > > > > > >>>>>>>> types. It's entirely reasonable to want a type
> > that
> > > > > allows
> > > > > > > NaN
> > > > > > > > > > >>>>> values,
> > > > > > > > > > >>>>>>> and
> > > > > > > > > > >>>>>>>> it's also reasonable for those of us building
> > > > databases,
> > > > > > > table
> > > > > > > > > > >>>>>> standards,
> > > > > > > > > > >>>>>>>> and vector indexes to care about guarantees that
> > > > > elements
> > > > > > > are
> > > > > > > > > > >>>> finite.
> > > > > > > > > > >>>>>>>> There's more than enough examples to show that
> > this
> > > > is a
> > > > > > > > > > >>> legitimate
> > > > > > > > > > >>>>> use
> > > > > > > > > > >>>>>>>> case.
> > > > > > > > > > >>>>>>>>
> > > > > > > > > > >>>>>>>> Since I don't see the concerns of the different
> > use
> > > > > cases
> > > > > > > > > > >>> changing,
> > > > > > > > > > >>>>> it
> > > > > > > > > > >>>>>>>> looks like we need to find a compromise to move
> > > > > forward. I
> > > > > > > > > > >>> propose
> > > > > > > > > > >>>>> that
> > > > > > > > > > >>>>>>> we
> > > > > > > > > > >>>>>>>> introduce a FINITE_VECTOR type and separately a
> > > > > > > > > > >> FIXED_LENGTH_LIST
> > > > > > > > > > >>>> or
> > > > > > > > > > >>>>>>>> another VECTOR type that allows infinite and NaN
> > > > values.
> > > > > > > > > > >>>>>>>>
> > > > > > > > > > >>>>>>>> Can we agree on a compromise? Is there a strong
> > reason
> > > > > to
> > > > > > > try
> > > > > > > > > > >> to
> > > > > > > > > > >>>> fit
> > > > > > > > > > >>>>>>> these
> > > > > > > > > > >>>>>>>> use cases into a single type? What if that means
> > not
> > > > > > making
> > > > > > > > > > >>>> progress?
> > > > > > > > > > >>>>>>>>
> > > > > > > > > > >>>>>>>> Ryan
> > > > > > > > > > >>>>>>>>
> > > > > > > > > > >>>>>>>> On Mon, Oct 5, 2026 at 12:15 PM Philipp
> Fischbeck
> > via
> > > > > dev
> > > > > > <
> > > > > > > > > > >>>>>>>> dev@parquet.apache.org> wrote:
> > > > > > > > > > >>>>>>>>
> > > > > > > > > > >>>>>>>>> Thinking about this more, I am also fine with
> > pulling
> > > > > out
> > > > > > > the
> > > > > > > > > > >>>>>> “NaN/Inf”
> > > > > > > > > > >>>>>>>>> requirement. Either into a type boolean (my
> > > > > preference),
> > > > > > or
> > > > > > > > > > >>>> putting
> > > > > > > > > > >>>>>> no
> > > > > > > > > > >>>>>>>>> restrictions on the Parquet level, allowing all
> > IEEE
> > > > > 754
> > > > > > > > > > >> float
> > > > > > > > > > >>>>>> numbers.
> > > > > > > > > > >>>>>>>> If
> > > > > > > > > > >>>>>>>>> some top-level application wants to put
> > additional
> > > > > > > > > > >> constraints
> > > > > > > > > > >>> on
> > > > > > > > > > >>>>> the
> > > > > > > > > > >>>>>>>> data
> > > > > > > > > > >>>>>>>>> type, like Iceberg not allowing NaN/infinity,
> it
> > can
> > > > > > still
> > > > > > > do
> > > > > > > > > > >>> so
> > > > > > > > > > >>>>> one
> > > > > > > > > > >>>>>>>> layer
> > > > > > > > > > >>>>>>>>> above. It seems like this was the biggest point
> > of
> > > > > > > discussion
> > > > > > > > > > >>>>>>> remaining.
> > > > > > > > > > >>>>>>>>> All these options work for me, and we should
> move
> > > > > forward
> > > > > > > > > > >> with
> > > > > > > > > > >>>> this
> > > > > > > > > > >>>>>>>>> proposal.
> > > > > > > > > > >>>>>>>>>
> > > > > > > > > > >>>>>>>>> Concerning additional element types like bf16,
> > they
> > > > can
> > > > > > be
> > > > > > > > > > >>>>> introduced
> > > > > > > > > > >>>>>>> as
> > > > > > > > > > >>>>>>>>> Parquet types (at least in the current
> LIST-based
> > > > spec
> > > > > > > > > > >>> proposal).
> > > > > > > > > > >>>>>>>>>
> > > > > > > > > > >>>>>>>>> Concerning Will’s suggestion around
> representing
> > > > VECTOR
> > > > > > as
> > > > > > > a
> > > > > > > > > > >>>>> variable
> > > > > > > > > > >>>>>>>>> length list with a MIN/MAX_LENGTH metadata
> > > > > restriction: I
> > > > > > > > > > >> think
> > > > > > > > > > >>>>> that
> > > > > > > > > > >>>>>> a
> > > > > > > > > > >>>>>>>>> dedicated logical VECTOR type still provides
> > enough
> > > > > > > benefits
> > > > > > > > > > >> by
> > > > > > > > > > >>>>>>>> describing
> > > > > > > > > > >>>>>>>>> the semantics of the data, and allowing to
> > capture
> > > > > future
> > > > > > > > > > >>> changes
> > > > > > > > > > >>>>>>> around
> > > > > > > > > > >>>>>>>>> statistics. In that case, the length is better
> > > > captured
> > > > > > as
> > > > > > > an
> > > > > > > > > > >>>> exact
> > > > > > > > > > >>>>>>>> length
> > > > > > > > > > >>>>>>>>> for this type specifically. I’m also not sure
> > about
> > > > > other
> > > > > > > use
> > > > > > > > > > >>>> cases
> > > > > > > > > > >>>>>> for
> > > > > > > > > > >>>>>>>>> MIN_LENGTH, so introducing a new type
> constraint
> > > > system
> > > > > > > just
> > > > > > > > > > >>> for
> > > > > > > > > > >>>>> the
> > > > > > > > > > >>>>>>>>> MIN_LENGTH=MAX_LENGTH case for vectors doesn’t
> > have
> > > > > > enough
> > > > > > > > > > >>> value
> > > > > > > > > > >>>> in
> > > > > > > > > > >>>>>> my
> > > > > > > > > > >>>>>>>>> view.
> > > > > > > > > > >>>>>>>>>
> > > > > > > > > > >>>>>>>>> Best,
> > > > > > > > > > >>>>>>>>> Philipp
> > > > > > > > > > >>>>>>>>>
> > > > > > > > > > >>>>>>>>>
> > > > > > > > > > >>>>>>>>> On Thu, Oct 1, 2026 at 9:00 AM Will Edwards via
> > dev <
> > > > > > > > > > >>>>>>>>> dev@parquet.apache.org>
> > > > > > > > > > >>>>>>>>> wrote:
> > > > > > > > > > >>>>>>>>>
> > > > > > > > > > >>>>>>>>>> +1 Sebastian, a vector type suggests usage
> > beyond
> > > > > > > > > > >> normalized
> > > > > > > > > > >>>>>>> embeddings
> > > > > > > > > > >>>>>>>>> and
> > > > > > > > > > >>>>>>>>>> engines really aren't going to be able to
> elide
> > all
> > > > > the
> > > > > > > > > > >>> checks
> > > > > > > > > > >>>>>>> anyway.
> > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > >>>>>>>>>> Antoine, you suggested we explore metadata
> > solutions
> > > > > too
> > > > > > > :)
> > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > >>>>>>>>>> Today, a database supporting CHAR(N) or
> > VARCHAR(n)
> > > > > can't
> > > > > > > > > > >>>> express
> > > > > > > > > > >>>>>> this
> > > > > > > > > > >>>>>>>> in
> > > > > > > > > > >>>>>>>>>> Parquet.  Engines can write opaque blobs they
> > > > > understand
> > > > > > > in
> > > > > > > > > > >>> the
> > > > > > > > > > >>>>>>>> metadata
> > > > > > > > > > >>>>>>>>>> (e.g. Arrow does) so the extra type info
> > > > round-trips,
> > > > > > but
> > > > > > > > > > >>> this
> > > > > > > > > > >>>>>> isn't
> > > > > > > > > > >>>>>>>>>> standardized, portable, nor interoperable.
> > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > >>>>>>>>>> So perhaps standardizing constraint metadata
> or
> > new
> > > > > > Thrift
> > > > > > > > > > >>>> fields
> > > > > > > > > > >>>>>> (my
> > > > > > > > > > >>>>>>>>> pref)
> > > > > > > > > > >>>>>>>>>> can generically support VECTOR(N) whilst also
> > being
> > > > > more
> > > > > > > > > > >>>> broadly
> > > > > > > > > > >>>>>>> useful
> > > > > > > > > > >>>>>>>>> for
> > > > > > > > > > >>>>>>>>>> other use cases and data types?
> > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > >>>>>>>>>> Imagine metadata or Thrift fields to tag the
> > > > > MIN_LENGTH
> > > > > > > and
> > > > > > > > > > >>>>>>> MAX_LENGTH
> > > > > > > > > > >>>>>>>>> for
> > > > > > > > > > >>>>>>>>>> Parquet's variable-length physcial types.
> These
> > > > would
> > > > > > > > > > >> count
> > > > > > > > > > >>>>>> logical
> > > > > > > > > > >>>>>>>>> units
> > > > > > > > > > >>>>>>>>>> (e.g. UTF8 code points for STRING) rather than
> > > > bytes.
> > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > >>>>>>>>>> Then one engine's VECTOR(N) can be written to
> > > > Parquet
> > > > > as
> > > > > > > > > > >>>> LIST(N)
> > > > > > > > > > >>>>>> with
> > > > > > > > > > >>>>>>>>>> MIN_LENGTH = MAX_LENGTH = N and then readers -
> > even
> > > > > > > > > > >> different
> > > > > > > > > > >>>>>>> engines -
> > > > > > > > > > >>>>>>>>> can
> > > > > > > > > > >>>>>>>>>> pick up these constraints if they support
> doing
> > so.
> > > > > We
> > > > > > > > > > >> don't
> > > > > > > > > > >>>>>>> actually
> > > > > > > > > > >>>>>>>>> need
> > > > > > > > > > >>>>>>>>>> a logical type to express a VECTOR then.
> > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > >>>>>>>>>> This also allows standard SQL's CHAR(N) and
> > > > VARCHAR(N)
> > > > > > to
> > > > > > > > > > >> be
> > > > > > > > > > >>>>>>> expressed
> > > > > > > > > > >>>>>>>> on
> > > > > > > > > > >>>>>>>>>> UTF8 columns etc.
> > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > >>>>>>>>>> This also seems generally useful in the future
> > as we
> > > > > > > > > > >> approach
> > > > > > > > > > >>>>>>>> expressing
> > > > > > > > > > >>>>>>>>>> other tensor data shapes and types.
> > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > >>>>>>>>>> This won't change the mechanics: the storage
> > type
> > > > > > doesn't
> > > > > > > > > > >>>> enforce
> > > > > > > > > > >>>>>>> these
> > > > > > > > > > >>>>>>>>>> constraints and engines must check them. It's
> > > > > previously
> > > > > > > > > > >>>>>> established
> > > > > > > > > > >>>>>>>> that
> > > > > > > > > > >>>>>>>>>> performance is just a software problem and
> > there is
> > > > no
> > > > > > > > > > >>>>> fundamental
> > > > > > > > > > >>>>>>>>>> performance advantage to the fixed size types
> in
> > > > > > > > > > >> Dremel-like
> > > > > > > > > > >>>>>> formats
> > > > > > > > > > >>>>>>>> like
> > > > > > > > > > >>>>>>>>>> Parquet.
> > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > >>>>>>>>>> Best,
> > > > > > > > > > >>>>>>>>>> Will
> > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > >>>>>>>>>> On Wed, 30 Sept 2026 at 16:08, Sebastian
> > Baunsgaard
> > > > <
> > > > > > > > > > >>>>>>>>> baunsgaard@apache.org
> > > > > > > > > > >>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>> wrote:
> > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>> After listening in on the last meeting
> > Wednesday, I
> > > > > > have
> > > > > > > > > > >>> been
> > > > > > > > > > >>>>>>>> thinking
> > > > > > > > > > >>>>>>>>>>> about this quite a bit, and thought I would
> > share
> > > > my
> > > > > 2
> > > > > > > > > > >>> cents.
> > > > > > > > > > >>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>> I suggest not requiring VECTOR elements to be
> > > > finite,
> > > > > > as
> > > > > > > > > > >>>>> proposed
> > > > > > > > > > >>>>>>> in
> > > > > > > > > > >>>>>>>>> the
> > > > > > > > > > >>>>>>>>>> PR
> > > > > > > > > > >>>>>>>>>>> <
> > https://github.com/apache/parquet-format/pull/624
> > > > >.
> > > > > > > > > > >>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>> While Divjot and others note the
> > specialization for
> > > > > > > > > > >> UTF-8,
> > > > > > > > > > >>>>>> Parquet
> > > > > > > > > > >>>>>>>>>> already
> > > > > > > > > > >>>>>>>>>>> defines DOUBLE and FLOAT as IEEE encodings
> > > > > > > > > > >>>>>>>>>>> <
> > > > > > > > > > >>>>>>>>
> > > > > > > > > > >>>>
> > > > > > > > >
> > > > >
> https://github.com/apache/parquet-format/blob/master/README.md#types
> > > > > > > > > > >>>>>>>>>> .
> > > > > > > > > > >>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>> Making NaN or +/-Inf illegal inside VECTOR
> > would
> > > > be a
> > > > > > new
> > > > > > > > > > >>>>>>> restriction
> > > > > > > > > > >>>>>>>>> on
> > > > > > > > > > >>>>>>>>>>> those same underlying physical types. Scalar
> > > > > > FLOAT/DOUBLE
> > > > > > > > > > >>>>> columns
> > > > > > > > > > >>>>>>>> have
> > > > > > > > > > >>>>>>>>>>> never had such a rule.
> > > > > > > > > > >>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>> It would also not match how the systems that
> > > > actually
> > > > > > > > > > >>> produce
> > > > > > > > > > >>>>>> these
> > > > > > > > > > >>>>>>>>>> vectors
> > > > > > > > > > >>>>>>>>>>> treat floats. Most of them use IEEE floats,
> > e.g.
> > > > > Spark,
> > > > > > > > > > >>>> DuckDB,
> > > > > > > > > > >>>>>>>> NumPy,
> > > > > > > > > > >>>>>>>>>>> PyTorch, TensorFlow, or lower level CUDA,
> > oneMKL,
> > > > and
> > > > > > > > > > >>>> OpenBLAS.
> > > > > > > > > > >>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>> pgvector's "all elements must be finite" rule
> > is an
> > > > > > > > > > >> insert
> > > > > > > > > > >>>>>> policy.
> > > > > > > > > > >>>>>>>>> While
> > > > > > > > > > >>>>>>>>>> I
> > > > > > > > > > >>>>>>>>>>> think Parquet as a general-purpose format
> > should
> > > > > allow
> > > > > > > > > > >> them
> > > > > > > > > > >>>> out
> > > > > > > > > > >>>>>> of
> > > > > > > > > > >>>>>>>> the
> > > > > > > > > > >>>>>>>>>> box.
> > > > > > > > > > >>>>>>>>>>> It is a user system decision to disallow
> > writing
> > > > > these
> > > > > > > > > > >>> values
> > > > > > > > > > >>>>> or
> > > > > > > > > > >>>>>> to
> > > > > > > > > > >>>>>>>>> throw
> > > > > > > > > > >>>>>>>>>>> errors when encountering them.
> > > > > > > > > > >>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>> A type-level ban also does not keep Inf/NaN
> > out of
> > > > > > vector
> > > > > > > > > > >>>> math.
> > > > > > > > > > >>>>>>>> Finite
> > > > > > > > > > >>>>>>>>>>> coordinates still overflow. For instance:
> > > > magnitude =
> > > > > > > > > > >>>>>> ||[FLOAT_MAX,
> > > > > > > > > > >>>>>>>>>>> FLOAT_MAX]|| where every coordinate is finite
> > > > > overflows
> > > > > > > > > > >> to
> > > > > > > > > > >>>>> +Inf.
> > > > > > > > > > >>>>>>>>> PyTorch
> > > > > > > > > > >>>>>>>>>>> documents the same overflow: tensor([1e20,
> > > > > > 1e20]).norm()
> > > > > > > > > > >> is
> > > > > > > > > > >>>> inf
> > > > > > > > > > >>>>>> in
> > > > > > > > > > >>>>>>>>> fp32.
> > > > > > > > > > >>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>> A generated MAGNITUDE(vec) column would then
> > be Inf
> > > > > > while
> > > > > > > > > > >>> the
> > > > > > > > > > >>>>>>> VECTOR
> > > > > > > > > > >>>>>>>>>> itself
> > > > > > > > > > >>>>>>>>>>> was required to be finite. The ban only
> forbids
> > > > > storing
> > > > > > > > > > >>>>> specials
> > > > > > > > > > >>>>>> in
> > > > > > > > > > >>>>>>>> the
> > > > > > > > > > >>>>>>>>>>> coordinates, and does not prevent even simple
> > > > > > downstream
> > > > > > > > > > >>>>>>> computations
> > > > > > > > > > >>>>>>>>>> from
> > > > > > > > > > >>>>>>>>>>> producing them.
> > > > > > > > > > >>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>> I suggest keeping VECTOR as the type and add
> an
> > > > > > optional
> > > > > > > > > > >>> bool
> > > > > > > > > > >>>>>>>>>>> VectorType.finite (default false). Engines
> that
> > > > want
> > > > > > > > > > >>>>>> pgvector-like
> > > > > > > > > > >>>>>>>>>>> rejection can require the flag = true.
> Engines
> > that
> > > > > use
> > > > > > > > > > >>>> NumPy /
> > > > > > > > > > >>>>>>>>> PyTorch /
> > > > > > > > > > >>>>>>>>>>> Spark arrays can keep representable IEEE
> > values.
> > > > For
> > > > > > the
> > > > > > > > > > >>> same
> > > > > > > > > > >>>>>>>> reason, I
> > > > > > > > > > >>>>>>>>>> do
> > > > > > > > > > >>>>>>>>>>> not favor adding yet another type called
> > > > > FINITE_VECTOR.
> > > > > > > > > > >>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>> Best,
> > > > > > > > > > >>>>>>>>>>> Sebastian
> > > > > > > > > > >>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>> On Tue, Sep 29, 2026 at 2:03 PM Divjot Arora
> > via
> > > > dev
> > > > > <
> > > > > > > > > > >>>>>>>>>>> dev@parquet.apache.org>
> > > > > > > > > > >>>>>>>>>>> wrote:
> > > > > > > > > > >>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>> I agree with moving forward with the
> proposal
> > and
> > > > > > > > > > >>> renaming
> > > > > > > > > > >>>>>> VECTOR
> > > > > > > > > > >>>>>>>> to
> > > > > > > > > > >>>>>>>>>>>> FINITE_VECTOR or something
> > > > > > > > > > >>>>>>>>>>>> similar if others feel that better
> > communicates
> > > > its
> > > > > > > > > > >>>> intent. I
> > > > > > > > > > >>>>>> see
> > > > > > > > > > >>>>>>>>> value
> > > > > > > > > > >>>>>>>>>>> in
> > > > > > > > > > >>>>>>>>>>>> having a vector type that guarantees
> > > > > > > > > > >>>>>>>>>>>> its values are mathematically-correct
> vectors
> > for
> > > > a
> > > > > > few
> > > > > > > > > > >>>>>> reasons:
> > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>> 1. As others have mentioned, this is similar
> > to
> > > > the
> > > > > > > > > > >>> STRING
> > > > > > > > > > >>>>>>> logical
> > > > > > > > > > >>>>>>>>> type
> > > > > > > > > > >>>>>>>>>>>> mandating UTF-8.
> > > > > > > > > > >>>>>>>>>>>> 2. It is not possible to rely on statistics
> > for
> > > > > > > > > > >> detecting
> > > > > > > > > > >>>> NaN
> > > > > > > > > > >>>>>> or
> > > > > > > > > > >>>>>>>>> +-inf
> > > > > > > > > > >>>>>>>>>>>> values as all statistics are optional.
> > > > > > > > > > >>>>>>>>>>>> To my knowledge, there is no precedent for
> > > > requiring
> > > > > > > > > > >> that
> > > > > > > > > > >>>>>> writers
> > > > > > > > > > >>>>>>>>> emit
> > > > > > > > > > >>>>>>>>>>>> stats for specific logical types.
> > > > > > > > > > >>>>>>>>>>>> 3. Without statistics, NaN/inf values could
> > flow
> > > > > > > > > > >> through
> > > > > > > > > > >>>> the
> > > > > > > > > > >>>>>>> engine
> > > > > > > > > > >>>>>>>>> and
> > > > > > > > > > >>>>>>>>>>>> block them for specific functionality.
> > > > > > > > > > >>>>>>>>>>>> This seems to be Lance's behavior. However,
> I
> > > > don't
> > > > > > > > > > >> think
> > > > > > > > > > >>>>> this
> > > > > > > > > > >>>>>> is
> > > > > > > > > > >>>>>>>> as
> > > > > > > > > > >>>>>>>>>> easy
> > > > > > > > > > >>>>>>>>>>>> as it sounds and Lance has run
> > > > > > > > > > >>>>>>>>>>>> into several bugs due to this ([1] [2] for
> > > > > examples).
> > > > > > > > > > >>> Their
> > > > > > > > > > >>>>>>> Python
> > > > > > > > > > >>>>>>>>> [3]
> > > > > > > > > > >>>>>>>>>>> and
> > > > > > > > > > >>>>>>>>>>>> Rust [4] APIs default to erroring on
> > > > > > > > > > >>>>>>>>>>>> vectors that contain NaN values.
> > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>> I agree that Parquet is a general-purpose
> > format,
> > > > > but
> > > > > > I
> > > > > > > > > > >>>> don't
> > > > > > > > > > >>>>>>> think
> > > > > > > > > > >>>>>>>>>> this
> > > > > > > > > > >>>>>>>>>>>> implies the lack of domain-specific types.
> > > > > > > > > > >>>>>>>>>>>> Instead, this broadens the use cases where
> > Parquet
> > > > > > > > > > >> would
> > > > > > > > > > >>> be
> > > > > > > > > > >>>>>>>>> considered
> > > > > > > > > > >>>>>>>>>> a
> > > > > > > > > > >>>>>>>>>>>> good fit. I also fully agree with
> > > > > > > > > > >>>>>>>>>>>> continuing work on a general fixed-size list
> > type
> > > > if
> > > > > > we
> > > > > > > > > > >>>>> believe
> > > > > > > > > > >>>>>>>> there
> > > > > > > > > > >>>>>>>>>> are
> > > > > > > > > > >>>>>>>>>>>> non-vector use cases that would benefit
> > > > > > > > > > >>>>>>>>>>>> from such guarantees at the storage layer.
> > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>> [1]
> > > > https://github.com/lance-format/lance/pull/3648
> > > > > > > > > > >>>>>>>>>>>> [2]
> > > > > https://github.com/lance-format/lance/issues/8607
> > > > > > > > > > >>> (see
> > > > > > > > > > >>>>>>> point 2
> > > > > > > > > > >>>>>>>>> in
> > > > > > > > > > >>>>>>>>>>>> "root
> > > > > > > > > > >>>>>>>>>>>> cause")
> > > > > > > > > > >>>>>>>>>>>> [3]
> > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > >>>>>>>>>
> > > > > > > > > > >>>>>>>>
> > > > > > > > > > >>>>>>>
> > > > > > > > > > >>>>>>
> > > > > > > > > > >>>>>
> > > > > > > > > > >>>>
> > > > > > > > > > >>>
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> >
> https://github.com/lancedb/lancedb/blob/0327618a62483feb6dca2336549845e5887cd48f/python/python/lancedb/table.py#L4807-L4911
> > > > > > > > > > >>>>>>>>>>>> [4]
> > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > >>>>>>>>>
> > > > > > > > > > >>>>>>>>
> > > > > > > > > > >>>>>>>
> > > > > > > > > > >>>>>>
> > > > > > > > > > >>>>>
> > > > > > > > > > >>>>
> > > > > > > > > > >>>
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> >
> https://github.com/lancedb/lancedb/blob/0327618a62483feb6dca2336549845e5887cd48f/rust/lancedb/src/table/add_data.rs#L105-L110
> > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>> On Mon, Sep 28, 2026 at 6:19 PM Philipp
> > Fischbeck
> > > > > via
> > > > > > > > > > >>> dev <
> > > > > > > > > > >>>>>>>>>>>> dev@parquet.apache.org> wrote:
> > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>> I also favor moving forward with the
> > proposal as
> > > > > is.
> > > > > > > > > > >> I
> > > > > > > > > > >>>> see
> > > > > > > > > > >>>>>>> strong
> > > > > > > > > > >>>>>>>>>> value
> > > > > > > > > > >>>>>>>>>>>> and
> > > > > > > > > > >>>>>>>>>>>>> use cases for the numeric vector type, and
> I
> > > > agree
> > > > > > > > > > >> with
> > > > > > > > > > >>>> the
> > > > > > > > > > >>>>>>>>>>> restrictions
> > > > > > > > > > >>>>>>>>>>>>> around NaN/Inf values and element types in
> > > > general.
> > > > > > > > > > >>>>> Precedent
> > > > > > > > > > >>>>>>>>> exists
> > > > > > > > > > >>>>>>>>>> in
> > > > > > > > > > >>>>>>>>>>>>> other types that restrict which physical
> type
> > > > > values
> > > > > > > > > > >>> are
> > > > > > > > > > >>>>>>>> logically
> > > > > > > > > > >>>>>>>>>>> valid
> > > > > > > > > > >>>>>>>>>>>>> (e.g., the geospatial types needing to be
> > > > > > > > > > >> WKB-encoded).
> > > > > > > > > > >>>> A
> > > > > > > > > > >>>>>>>> general
> > > > > > > > > > >>>>>>>>>>>>> fixed-size list also has value, but I think
> > it
> > > > > makes
> > > > > > > > > > >>>> sense
> > > > > > > > > > >>>>> to
> > > > > > > > > > >>>>>>>>>> separate
> > > > > > > > > > >>>>>>>>>>>> the
> > > > > > > > > > >>>>>>>>>>>>> two types. The semantics are relevant, and
> I
> > can
> > > > > see
> > > > > > > > > > >>> how
> > > > > > > > > > >>>>> the
> > > > > > > > > > >>>>>>> two
> > > > > > > > > > >>>>>>>>>> types
> > > > > > > > > > >>>>>>>>>>>>> might diverge further in the future
> regarding
> > > > > > > > > > >>> statistics
> > > > > > > > > > >>>> or
> > > > > > > > > > >>>>>>>>>> encodings.
> > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>> I would be okay with renaming the vector
> > type,
> > > > but
> > > > > it
> > > > > > > > > > >>>> seems
> > > > > > > > > > >>>>>>> like
> > > > > > > > > > >>>>>>>>> the
> > > > > > > > > > >>>>>>>>>>>> VECTOR
> > > > > > > > > > >>>>>>>>>>>>> name is already used across other engines
> > > > mentioned
> > > > > > > > > > >> in
> > > > > > > > > > >>>> this
> > > > > > > > > > >>>>>>>> thread.
> > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>> Best,
> > > > > > > > > > >>>>>>>>>>>>> Philipp
> > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>> On Sat, Sep 26, 2026 at 4:37 PM Gang Wu <
> > > > > > > > > > >>>> ustcwg@gmail.com>
> > > > > > > > > > >>>>>>>> wrote:
> > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>> To separate orthogonal concerns, I feel we
> > > > should
> > > > > > > > > > >>> treat
> > > > > > > > > > >>>>> the
> > > > > > > > > > >>>>>>>>> general
> > > > > > > > > > >>>>>>>>>>>>>> FixedSizeList as the underlying physical
> > layout,
> > > > > > > > > > >> and
> > > > > > > > > > >>>>>> separate
> > > > > > > > > > >>>>>>>> it
> > > > > > > > > > >>>>>>>>>> from
> > > > > > > > > > >>>>>>>>>>>> the
> > > > > > > > > > >>>>>>>>>>>>>> discussion of a specialized VECTOR (or
> > broader
> > > > ML
> > > > > > > > > > >>>> tensor)
> > > > > > > > > > >>>>>>>> logical
> > > > > > > > > > >>>>>>>>>>> type.
> > > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>> Currently, the immediate need is to
> define a
> > > > > > > > > > >>>> specialized
> > > > > > > > > > >>>>>>> VECTOR
> > > > > > > > > > >>>>>>>>>> type
> > > > > > > > > > >>>>>>>>>>> so
> > > > > > > > > > >>>>>>>>>>>>>> that different engines and formats have a
> > common
> > > > > > > > > > >>>> semantic
> > > > > > > > > > >>>>>>>>> contract
> > > > > > > > > > >>>>>>>>>>> for
> > > > > > > > > > >>>>>>>>>>>>>> interoperability.
> > > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>> If the main point of contention is just
> the
> > > > > > > > > > >>>> NaN/Inf/null
> > > > > > > > > > >>>>>>>>>> limitations,
> > > > > > > > > > >>>>>>>>>>>>> maybe
> > > > > > > > > > >>>>>>>>>>>>>> we can resolve this by adding a
> > qualification to
> > > > > > > > > > >> the
> > > > > > > > > > >>>> name
> > > > > > > > > > >>>>>>> (like
> > > > > > > > > > >>>>>>>>>>> Andrew
> > > > > > > > > > >>>>>>>>>>>>>> suggested, e.g., VECTOR_EMBEDDING or
> > > > > FINITE_VECTOR)
> > > > > > > > > > >>> to
> > > > > > > > > > >>>>>> avoid
> > > > > > > > > > >>>>>>>>>>> confusion
> > > > > > > > > > >>>>>>>>>>>>> with
> > > > > > > > > > >>>>>>>>>>>>>> broader ML use cases. Alternatively, we
> > could
> > > > > > > > > > >>>> introduce a
> > > > > > > > > > >>>>>>>> boolean
> > > > > > > > > > >>>>>>>>>>> flag
> > > > > > > > > > >>>>>>>>>>>>>> (e.g., optional bool VectorType.finite) as
> > > > > proposed
> > > > > > > > > > >>>>>> earlier.
> > > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>> On Fri, Sep 25, 2026 at 3:52 AM Andrew
> Lamb
> > <
> > > > > > > > > > >>>>>>>>>> andrewlamb11@gmail.com>
> > > > > > > > > > >>>>>>>>>>>>>> wrote:
> > > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>>> I am a little confused about the points
> of
> > > > > > > > > > >>>> disagreement
> > > > > > > > > > >>>>>>> here.
> > > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>>> Is it that the name `VECTOR` is too
> > general for
> > > > > > > > > > >> the
> > > > > > > > > > >>>>>>> specific
> > > > > > > > > > >>>>>>>>> use
> > > > > > > > > > >>>>>>>>>>> case
> > > > > > > > > > >>>>>>>>>>>>> of
> > > > > > > > > > >>>>>>>>>>>>>>> fixed length arrays with non-finite
> values
> > used
> > > > > > > > > > >> for
> > > > > > > > > > >>>>>> vector
> > > > > > > > > > >>>>>>>>>>>> embeddings?
> > > > > > > > > > >>>>>>>>>>>>>> If
> > > > > > > > > > >>>>>>>>>>>>>>> so, maybe we could add a qualification to
> > the
> > > > > > > > > > >> name
> > > > > > > > > > >>>>>>>>>> (FINITE_VECTOR?
> > > > > > > > > > >>>>>>>>>>>>>>> FEATURE_VECTOR? VECTOR_EMBEDDING?)
> > > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>>> It seems like there is precedent for
> > defining
> > > > > > > > > > >>> logical
> > > > > > > > > > >>>>>> types
> > > > > > > > > > >>>>>>>>> that
> > > > > > > > > > >>>>>>>>>>>>> specify
> > > > > > > > > > >>>>>>>>>>>>>>> certain physical types and restrict their
> > > > > > > > > > >> contents
> > > > > > > > > > >>> to
> > > > > > > > > > >>>>>> help
> > > > > > > > > > >>>>>>>>>> readers
> > > > > > > > > > >>>>>>>>>>>>>>> interpret meaning, for example string[1]
> or
> > > > JSON;
> > > > > > > > > > >>>>>> Defining
> > > > > > > > > > >>>>>>>> one
> > > > > > > > > > >>>>>>>>>> for
> > > > > > > > > > >>>>>>>>>>>>>> vector
> > > > > > > > > > >>>>>>>>>>>>>>> embeddings seems reasonable enough to me.
> > > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>>> Andrew
> > > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>>> [1]:
> > > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > >>>>>>>>>
> > > > > > > > > > >>>>>>>>
> > > > > > > > > > >>>>>>>
> > > > > > > > > > >>>>>>
> > > > > > > > > > >>>>>
> > > > > > > > > > >>>>
> > > > > > > > > > >>>
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> >
> https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#string
> > > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>>> On Thu, Sep 24, 2026 at 3:34 PM Daniel
> > Weeks <
> > > > > > > > > > >>>>>>>>> dweeks@apache.org>
> > > > > > > > > > >>>>>>>>>>>>> wrote:
> > > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>>>>> Well, Parquet is not a single-purpose
> > format.
> > > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>>>>> Vector stores can choose whatever
> policy
> > fits
> > > > > > > > > > >>>> their
> > > > > > > > > > >>>>>>>> usage,
> > > > > > > > > > >>>>>>>>>> but
> > > > > > > > > > >>>>>>>>>>>>> we're
> > > > > > > > > > >>>>>>>>>>>>>>>>> talking about a general-purpose file
> > format
> > > > > > > > > > >>> that
> > > > > > > > > > >>>>> aims
> > > > > > > > > > >>>>>>> to
> > > > > > > > > > >>>>>>>> be
> > > > > > > > > > >>>>>>>>>>>> broadly
> > > > > > > > > > >>>>>>>>>>>>>>>>> applicable.
> > > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>>>> I agree that the format is intended to
> be
> > > > > > > > > > >> general
> > > > > > > > > > >>>>>>> purpose,
> > > > > > > > > > >>>>>>>>> but
> > > > > > > > > > >>>>>>>>>>>> types
> > > > > > > > > > >>>>>>>>>>>>>>> within
> > > > > > > > > > >>>>>>>>>>>>>>>> the format define specific
> representation
> > and
> > > > > > > > > > >>>>> semantics
> > > > > > > > > > >>>>>>> to
> > > > > > > > > > >>>>>>>>>>> achieve
> > > > > > > > > > >>>>>>>>>>>>> that
> > > > > > > > > > >>>>>>>>>>>>>>>> goal.
> > > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>>>>> "Vector" has a more general sense than
> > ML and
> > > > > > > > > > >>>> it's
> > > > > > > > > > >>>>>>> quite
> > > > > > > > > > >>>>>>>>>>> trivial
> > > > > > > > > > >>>>>>>>>>>> to
> > > > > > > > > > >>>>>>>>>>>>>>>>> produce vectors with non-finite values
> > using
> > > > > > > > > > >>> e.g.
> > > > > > > > > > >>>>>> NumPy
> > > > > > > > > > >>>>>>>> or
> > > > > > > > > > >>>>>>>>>>> Pandas
> > > > > > > > > > >>>>>>>>>>>>> (or
> > > > > > > > > > >>>>>>>>>>>>>>>>> any other piece of dataframe or
> > scientific
> > > > > > > > > > >>>>> computing
> > > > > > > > > > >>>>>>>>>> software).
> > > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>>>> The justification for restricting
> > non-finite
> > > > > > > > > > >>>> values
> > > > > > > > > > >>>>> is
> > > > > > > > > > >>>>>>>>>> directly
> > > > > > > > > > >>>>>>>>>>>>>> related
> > > > > > > > > > >>>>>>>>>>>>>>> to
> > > > > > > > > > >>>>>>>>>>>>>>>> the semantic meaning of a vector.
> Vectors
> > > > have
> > > > > > > > > > >>>>>> specific
> > > > > > > > > > >>>>>>>>>>>>>> characteristics
> > > > > > > > > > >>>>>>>>>>>>>>>> (dimension, magnitude, direction, etc).
> > > > > > > > > > >>> Non-finite
> > > > > > > > > > >>>>>>> values
> > > > > > > > > > >>>>>>>>> are
> > > > > > > > > > >>>>>>>>>>>>> invalid
> > > > > > > > > > >>>>>>>>>>>>>>> and
> > > > > > > > > > >>>>>>>>>>>>>>>> break those characteristics.  I'd liken
> > it to
> > > > > > > > > > >>>> storing
> > > > > > > > > > >>>>>>>>> non-utf8
> > > > > > > > > > >>>>>>>>>>>> values
> > > > > > > > > > >>>>>>>>>>>>>> in
> > > > > > > > > > >>>>>>>>>>>>>>> a
> > > > > > > > > > >>>>>>>>>>>>>>>> string and still calling it a string.
> > > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>>>> That also differentiates a vector from a
> > fixed
> > > > > > > > > > >>>> length
> > > > > > > > > > >>>>>>> list,
> > > > > > > > > > >>>>>>>>>> which
> > > > > > > > > > >>>>>>>>>>>> is
> > > > > > > > > > >>>>>>>>>>>>>> more
> > > > > > > > > > >>>>>>>>>>>>>>>> general and does not carry the same
> > semantic
> > > > > > > > > > >>>> meaning.
> > > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>>>> I also don't think we need to bundle all
> > > > > > > > > > >>> similarly
> > > > > > > > > > >>>>>> shaped
> > > > > > > > > > >>>>>>>> use
> > > > > > > > > > >>>>>>>>>>> cases
> > > > > > > > > > >>>>>>>>>>>>>> into
> > > > > > > > > > >>>>>>>>>>>>>>>> the same solution (e.g. flxed size
> list).
> > > > > > > > > > >> Vector
> > > > > > > > > > >>>> is
> > > > > > > > > > >>>>> a
> > > > > > > > > > >>>>>>> well
> > > > > > > > > > >>>>>>>>>>>>> established
> > > > > > > > > > >>>>>>>>>>>>>>>> data type at this point and aligns with
> > the
> > > > > > > > > > >>> general
> > > > > > > > > > >>>>>>> purpose
> > > > > > > > > > >>>>>>>>>>> nature
> > > > > > > > > > >>>>>>>>>>>> of
> > > > > > > > > > >>>>>>>>>>>>>> the
> > > > > > > > > > >>>>>>>>>>>>>>>> format (many other formats and databases
> > > > > > > > > > >> natively
> > > > > > > > > > >>>>>> support
> > > > > > > > > > >>>>>>>>>> this).
> > > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>>>> -Dan
> > > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>>>> On Thu, Sep 24, 2026 at 11:55 AM Antoine
> > > > > > > > > > >> Pitrou <
> > > > > > > > > > >>>>>>>>>>>> antoine@python.org>
> > > > > > > > > > >>>>>>>>>>>>>>>> wrote:
> > > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>>>>> Le 24/09/2026 à 20:11, Russell Spitzer
> a
> > > > > > > > > > >> écrit
> > > > > > > > > > >>> :
> > > > > > > > > > >>>>>>>>>>>>>>>>>> I think something I haven't seen yet
> is
> > the
> > > > > > > > > > >>>> clear
> > > > > > > > > > >>>>>>>>> use-case
> > > > > > > > > > >>>>>>>>>>> for
> > > > > > > > > > >>>>>>>>>>>>>>>> non-finite
> > > > > > > > > > >>>>>>>>>>>>>>>>>> values within a vector type. There has
> > > > > > > > > > >> been a
> > > > > > > > > > >>>> lot
> > > > > > > > > > >>>>>> of
> > > > > > > > > > >>>>>>>>>>>> theoretical
> > > > > > > > > > >>>>>>>>>>>>>>>>> discussion
> > > > > > > > > > >>>>>>>>>>>>>>>>>> here about use cases but what toolings
> > > > > > > > > > >>>> currently
> > > > > > > > > > >>>>>>>> produce
> > > > > > > > > > >>>>>>>>>>> these
> > > > > > > > > > >>>>>>>>>>>>>>> vectors
> > > > > > > > > > >>>>>>>>>>>>>>>>> and
> > > > > > > > > > >>>>>>>>>>>>>>>>>> how do they function in downstream
> > engines?
> > > > > > > > > > >>> I'd
> > > > > > > > > > >>>>>> like
> > > > > > > > > > >>>>>>> to
> > > > > > > > > > >>>>>>>>>> limit
> > > > > > > > > > >>>>>>>>>>>>>>>> introducing
> > > > > > > > > > >>>>>>>>>>>>>>>>>> types that have a wide range of values
> > > > > > > > > > >> which
> > > > > > > > > > >>>> are
> > > > > > > > > > >>>>>>> simply
> > > > > > > > > > >>>>>>>>> are
> > > > > > > > > > >>>>>>>>>>>>> failure
> > > > > > > > > > >>>>>>>>>>>>>>>> modes
> > > > > > > > > > >>>>>>>>>>>>>>>>>> of different flavors
> > > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>>>>> "Vector" has a more general sense than
> > ML and
> > > > > > > > > > >>>> it's
> > > > > > > > > > >>>>>>> quite
> > > > > > > > > > >>>>>>>>>>> trivial
> > > > > > > > > > >>>>>>>>>>>> to
> > > > > > > > > > >>>>>>>>>>>>>>>>> produce vectors with non-finite values
> > using
> > > > > > > > > > >>> e.g.
> > > > > > > > > > >>>>>> NumPy
> > > > > > > > > > >>>>>>>> or
> > > > > > > > > > >>>>>>>>>>> Pandas
> > > > > > > > > > >>>>>>>>>>>>> (or
> > > > > > > > > > >>>>>>>>>>>>>>>>> any other piece of dataframe or
> > scientific
> > > > > > > > > > >>>>> computing
> > > > > > > > > > >>>>>>>>>> software).
> > > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>>>>> It's not like the general concept of a
> > vector
> > > > > > > > > > >>> is
> > > > > > > > > > >>>> a
> > > > > > > > > > >>>>>> IETF
> > > > > > > > > > >>>>>>>>>>> standard
> > > > > > > > > > >>>>>>>>>>>> or
> > > > > > > > > > >>>>>>>>>>>>>>>>> anything, so analogies with JSON or
> UTF8
> > > > > > > > > > >> (where
> > > > > > > > > > >>>> the
> > > > > > > > > > >>>>>>>> values
> > > > > > > > > > >>>>>>>>>> are
> > > > > > > > > > >>>>>>>>>>>>>> expected
> > > > > > > > > > >>>>>>>>>>>>>>>>> to be valid according to the respective
> > > > > > > > > > >> specs)
> > > > > > > > > > >>> do
> > > > > > > > > > >>>>> not
> > > > > > > > > > >>>>>>>>> apply.
> > > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>>>>> This discussion would IMHO have a
> better
> > > > > > > > > > >> chance
> > > > > > > > > > >>>> of
> > > > > > > > > > >>>>>>>>> converging
> > > > > > > > > > >>>>>>>>>>> if
> > > > > > > > > > >>>>>>>>>>>> we
> > > > > > > > > > >>>>>>>>>>>>>>>>> separated the two orthogonal concerns
> of:
> > > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>>>>> 1. representing fixed-length
> list/vector
> > data
> > > > > > > > > > >>> in
> > > > > > > > > > >>>>>>> Parquet
> > > > > > > > > > >>>>>>>>>> (with
> > > > > > > > > > >>>>>>>>>>> a
> > > > > > > > > > >>>>>>>>>>>>>>>>> potential limitation to non-nested
> types)
> > > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>>>>> 2. representing the information that a
> > given
> > > > > > > > > > >>>> column
> > > > > > > > > > >>>>>> (or
> > > > > > > > > > >>>>>>>>>> column
> > > > > > > > > > >>>>>>>>>>>>> chunk,
> > > > > > > > > > >>>>>>>>>>>>>>>>> etc.) does not contain non-finite
> values
> > > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>>>>> ... both of which are useful on their
> own
> > > > > > > > > > >>> right,
> > > > > > > > > > >>>>> but
> > > > > > > > > > >>>>>> do
> > > > > > > > > > >>>>>>>> not
> > > > > > > > > > >>>>>>>>>>> have
> > > > > > > > > > >>>>>>>>>>>> to
> > > > > > > > > > >>>>>>>>>>>>>> be
> > > > > > > > > > >>>>>>>>>>>>>>>>> conflated and bundled together.
> > > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>>>>> Regards
> > > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>>>>> Antoine.
> > > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > >>>>>>>>>
> > > > > > > > > > >>>>>>>>
> > > > > > > > > > >>>>>>>
> > > > > > > > > > >>>>>>
> > > > > > > > > > >>>>>
> > > > > > > > > > >>>>
> > > > > > > > > > >>>
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> >
>

Re: [DISCUSS] Numeric Vector type: physical representation (outcome of focused call)

[Ryan Blue <bl...@apache.org>]

Thanks, Curt. I wasn't aware of that behavior in MySQL. I was trying to
learn more and I don't see it in documentation either. Do you have any more
information about how MySQL handles this? One page makes it sound like it
may be platform dependent
<https://dev.mysql.com/doc/refman/26.7/en/problems-with-float.html>, where
some values produce "inf" and some produce "0".

My original comment was referring to databases that allow users to specify
types that exclude non-finite values, which is what we would need in order
to support a finite float in table formats since we already support the
full range of values. If you know of any cases like that rather than just
limiting values, I'm curious. Thanks!

Ryan

On Wed, Oct 21, 2026 at 3:48 PM Curt Hagenlocher <cu...@hagenlocher.org>
wrote:

> > I'm not aware of databases with finite floating point types.
>
> Microsoft SQL Server, MySQL and SAP ASE (née Sybase) do not allow
> storing non-finite floating-point values in their binary
> floating-point columns. They don't have a separate type which does
> allow this.
>
> On Wed, Oct 21, 2026 at 3:30 PM Ryan Blue <bl...@apache.org> wrote:
> >
> > Jan,
> >
> > I agree that we _could_ model vectors as an orthogonal finiteness
> property
> > for elements in a fixed-length list. I don't know how useful that would
> be.
> > I've not had anyone ask for that, and I'm not aware of databases with
> > finite floating point types. I think it makes more sense to move forward
> > with the existing FINITE_VECTOR that is proposed.
> >
> > For the logical type argument, it sounds like the issue is that you want
> to
> > fall back to readers treating vectors as LIST. I don't see a good way to
> > make that happen, but it also isn't needed by the table format use case
> > because we are introducing this as a new type. Do you have a use case
> where
> > you are going to mix FINITE_VECTOR with LIST files and this will cause a
> > problem?
> >
> > I don't think that we've had an expectation for logical types in the past
> > that they produce a particular in-memory representation. Maybe we could
> > allow setting both `LIST` and `FINITE_VECTOR` logical types at the same
> > time? This kind of compatibility hasn't been a priority as far as I'm
> > aware, but if it is important to you we can consider it.
> >
> > Ryan
> >
> > On Wed, Oct 21, 2026 at 1:13 PM Jan Finis <jp...@gmail.com> wrote:
> >
> > >  assuming what will be proposed is a new logical type FINITE_VECTOR
> that
> > > annotates the same column that would have been annotated as LIST
> before:
> > >
> > > What I find somewhat suboptimal of FINITE_VECTOR is that finiteness is
> part
> > > of the vector type. Isn't the finiteness rather a property of the
> contained
> > > floats? Such a modeling would be more composable, as it would for
> example
> > > also allow to have normal (non-vector) float columns that are
> restricted to
> > > be finite. So, if we do think that we want finiteness in the type
> system, I
> > > would rather cut it into two parts: A fixed size vector concept, and a
> new
> > > way of labeling float columns finite. The vector type would then just
> boil
> > > down to be a fixed size list / we could just give the list type an
> optional
> > > fixed size field. Then, the column contained within it would be a float
> > > tagged as finite.
> > >
> > > It was mentioned that a logical type annotating a list of float would
> be
> > > forward compatible, but - depending on the modeling - that is not
> > > necessarily true: as logical types in Parquet do not support
> subtyping, a
> > > column cannot be a list of float and a (finite) vector at the same
> time,
> > > but only one of the two. It depends on how the type will be modeled in
> > > thrift in the end, but if it is a new union member of `LogicalType`,
> then
> > > legacy engines will not understand that such a vector can also just be
> > > interpreted as a list of floats. They will just see a nested column
> type of
> > > a logical type they don't understand, so they will likely reject
> reading
> > > this column (at least that's what the engine I'm maintaining would be
> > > doing).
> > >
> > > In contrast, if we just give the list type an optional fixed size and
> float
> > > an optional finiteness, then the resulting thrift is actually fully
> forward
> > > compatible: Legacy engines will just ignore the fixed length and treat
> the
> > > list as variable length and they will ignore the finiteness and treat
> the
> > > element type as float.
> > >
> > > (A side note, which most people here may already be aware of, but it's
> good
> > > to repeat it: The not-NULL property doesn't need new type support at
> all.
> > > The normal type system of list already allows setting the contained
> element
> > > type to be optional or required. If it is set to be required, then
> NULLs
> > > are not allowed.)
> > >
> > > Cheers,
> > > Jan
> > >
> > >
> > >
> > >
> > >
> > > Am Di., 20. Okt. 2026 um 20:37 Uhr schrieb Russell Spitzer <
> > > russell.spitzer@gmail.com>:
> > >
> > > > I support proceeding with the two types. I honestly think the cost of
> > > > adding new logical types is overestimated. If we have as many
> interested
> > > > parties as have appeared in this thread we already know there is a
> large
> > > > group of engineers ready to do the work and engines ready to consume
> (and
> > > > produce) the type. The fact that implementations currently use
> > > heuristics,
> > > > sampling (I loved Gunnar's talk on his hardwood work on this at CoC
> > > > Glasgow) or custom annotations to achieve the same performance
> benefits
> > > > makes me think we should just create the type and move on.
> > > >
> > > > On Mon, Oct 19, 2026 at 7:21 PM Ryan Blue <bl...@apache.org> wrote:
> > > >
> > > > > An annotation on list<float> is useful because:
> > > > > - We want to correctly annotate what is being stored
> > > > > - We want to unlock optimizations when each non-null list has the
> same
> > > > > number of elements
> > > > > - We want to have writer guarantees
> > > > >
> > > > > I don't think that logical type compatibility is a good argument.
> We
> > > want
> > > > > to avoid a case where we have to treat all binary like a string.
> And
> > > > > because we are introducing this in table formats after
> standardizing it
> > > > > here, we can do that reliably.
> > > > >
> > > > > As I said in my last email, I think there's a difference between
> > > > finiteness
> > > > > and general "invariants" you might want. Vectors aren't vectors if
> they
> > > > > don't have a length and a direction. Other invariants don't work
> the
> > > same
> > > > > way. A zero-length vector is still a valid vector and can be used,
> just
> > > > not
> > > > > for certain distance metrics.
> > > > >
> > > > > I also want to clarify that when we looked into this more closely
> and
> > > > > recognized the mismatch between the two separate use cases, I
> offered
> > > to
> > > > > split out the finite vector work so that we didn't cause any
> headaches
> > > > for
> > > > > people who are interested in fixed-length list. It sounds like not
> > > > > splitting out the use cases is causing a lot of headache. Vectors
> for
> > > our
> > > > > use case need a finiteness guarantee at write time, and
> fixed-length
> > > > lists
> > > > > need to store NaN and other values.
> > > > >
> > > > > It's okay to have two types and we should for two different use
> cases.
> > > > > Otherwise, we are forcing one to make an unnecessary compromise. I
> > > don't
> > > > > want to force everyone to exclude NaN and infinite values from
> > > > fixed-length
> > > > > lists. Seems reasonable that we should also be able to have vectors
> > > that
> > > > > exclude those values, since that is how they are used and we have
> to
> > > also
> > > > > guarantee it for exchanging data with other existing databases.
> > > > >
> > > > > Ryan
> > > > >
> > > > > On Mon, Oct 19, 2026 at 3:12 AM Will Edwards via dev <
> > > > > dev@parquet.apache.org>
> > > > > wrote:
> > > > >
> > > > > > It reads well. Three small fixes:
> > > > > >
> > > > > > 1. **"back in May or so"** is ambiguous. The
> > > > fixed-size-list-for-tensors
> > > > > > framing is from Rok's opening post in May 2024. The May 2026
> design
> > > doc
> > > > > was
> > > > > > already mostly about performance and layout. "Originally" avoids
> > > anyone
> > > > > > correcting the date.
> > > > > > 2. **"Spark's sparse feature vectors routinely use NaN"** isn't
> quite
> > > > > > right. The NaN-for-missing behaviour comes from
> `VectorAssembler` and
> > > > > > `Imputer` and applies to Spark ML vectors generally, dense or
> sparse.
> > > > > > "Spark's feature vectors, dense and sparse," fixes it.
> > > > > > 3. There are typos: "Iirc" should be "IIRC", and "Spark,
> BigQuery and
> > > > > > DuckDB and pretty much…" has a doubled "and".
> > > > > >
> > > > > > The last sentence of the invariants paragraph could be clearer.
> > > > "Although
> > > > > > that doesn't mean readers shouldn't check on read" has a double
> > > > negative
> > > > > > and is easy to misread. "Though careful readers will still check
> on
> > > > read"
> > > > > > is simpler.
> > > > > >
> > > > > > With those applied:
> > > > > >
> > > > > > ---
> > > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > The scope of this discussion keeps moving :). IIRC it originally
> was
> > > > > about
> > > > > > fixed-size lists for ML tensors, became a performance question,
> and
> > > is
> > > > > now
> > > > > > about whether vectors are mathematically valid. Each step made
> sense,
> > > > > but I
> > > > > > don't think we have consensus on where we've ended up?
> > > > > >
> > > > > > I think there is much useful work in what we left behind. Many ML
> > > > vectors
> > > > > > aren't embeddings for search. E.g. Spark's feature vectors,
> dense and
> > > > > > sparse, routinely use NaN to mark missing values, and quantised
> > > formats
> > > > > > pack codes with scales in ways Parquet can't describe today.
> > > > ML-oriented
> > > > > > formats like Lance are moving towards tensors, quantisation and
> > > > indexes.
> > > > > A
> > > > > > type that told engines how to interpret that kind of data would
> > > enable
> > > > > new
> > > > > > things. An annotation on LIST<FLOAT> doesn't.
> > > > > >
> > > > > > My big concern with the current focus on embeddings needing a
> logical
> > > > > type
> > > > > > is compatibility. Today LIST<FLOAT> is the common format for
> > > > embeddings,
> > > > > > and Spark, BigQuery, DuckDB and pretty much every other library
> I can
> > > > > think
> > > > > > of offer distance functions on plain lists, so any engine can
> use any
> > > > > file.
> > > > > > If engines start requiring a VECTOR annotation, existing files
> and
> > > > > anything
> > > > > > rewritten by a tool that doesn't know the type stop working. If
> they
> > > > > don't
> > > > > > require it, the annotation adds nothing. We've seen this before
> with
> > > > > > strings written without the UTF8 annotation, which engines still
> have
> > > > to
> > > > > > cope with years later.
> > > > > >
> > > > > > On invariants, I agree some users may want the writer to check
> what
> > > > they
> > > > > > write as they write it, but that can be a writer option rather
> than a
> > > > > type.
> > > > > > Readers that want evidence could get it from extra statistics or
> > > > > metadata,
> > > > > > though careful readers will still check on read.
> > > > > >
> > > > > > If a finite fixed-length vector contract is useful to table
> formats,
> > > > > > perhaps Iceberg is the place for it, with Parquet storing plain
> > > lists.
> > > > > I'd
> > > > > > raise the same worry there, though: if tables start rejecting
> > > untagged
> > > > > > columns, we've just moved the split up a layer.
> > > > > >
> > > > > > Best,
> > > > > > Will
> > > > > >
> > > > > > On Sat, 17 Oct 2026 at 00:10, Ryan Blue <bl...@apache.org> wrote:
> > > > > >
> > > > > > > I agree with much of what Dan said, but I'll add some to this
> > > thread
> > > > as
> > > > > > > well.
> > > > > > >
> > > > > > > First, I want to address the arguments that were pointed out:
> > > > > > > 1. Other vector databases have finiteness requirements
> > > > > > > 2. "Vectors are the hot stuff right now" -- let's jump on the
> > > > bandwagon
> > > > > > > 3. A general solution will take too long, finiteness will make
> > > faster
> > > > > > > progress
> > > > > > > 4. To give a guarantee to readers that no INF/NaN/NULL will be
> in
> > > the
> > > > > > > vector
> > > > > > >
> > > > > > > I don't think these represent our actual motivation or
> position,
> > > > which
> > > > > > > makes them function as a straw-man. This is probably not what
> was
> > > > > > > intended, but I want to make sure that the misunderstanding
> > > > > > > doesn't continue. For one thing, we aren't saying that we need
> > > finite
> > > > > > > vectors because they are "hot stuff".
> > > > > > >
> > > > > > > We want to build tools that use them on top of Parquet. I
> think it
> > > > > should
> > > > > > > also be clear that we aren't advocating for finite elements
> blindly
> > > > > just
> > > > > > > because other engines do (cargo cult design!). There are real
> > > > > > > interoperability issues if we choose to go a different
> direction so
> > > > > this
> > > > > > is
> > > > > > > a "Chesterton's Fence" situation. It's important to understand
> why
> > > > > > > those databases
> > > > > > > made that choice. I've also pointed this out as confirmation
> that
> > > > we're
> > > > > > > going down a reasonable and established path.
> > > > > > >
> > > > > > > The third argument sounds like a simple misunderstanding. We're
> > > > > pursuing
> > > > > > a
> > > > > > > logical type right now because that's forward compatible and we
> > > want
> > > > to
> > > > > > > label the data properly and set expectations for the type.
> Physical
> > > > > type
> > > > > > > changes break forward compatibility and need to be released in
> a
> > > new
> > > > > > format
> > > > > > > version . I think this is the source of the confusion since it
> has
> > > > been
> > > > > > in
> > > > > > > the same discussion, but the choice about finiteness is
> separate.
> > > > > > >
> > > > > > > That brings us to the last argument, whether there is value in
> > > > writers
> > > > > > > participating in guarantees. I think there's value in adding
> > > > > requirements
> > > > > > > for writers. It's up to readers how they want to use those
> > > > > requirements,
> > > > > > > but a writer requirement allows a policy of "garbage in garbage
> > > out"
> > > > > > rather
> > > > > > > than being forced to handle bad data. Why would a reader need
> to
> > > run
> > > > > > > validation on every element just in case a writer had a bug?
> And
> > > why
> > > > > > should
> > > > > > > the potential for writer bugs mean that it has to be the
> reader's
> > > > > > problem?
> > > > > > > If a writer produces bad data, we can absolutely reject it and
> tell
> > > > the
> > > > > > > writer to fix the bug.
> > > > > > >
> > > > > > > I'm also puzzled by the assertion that writer guarantees have
> > > value,
> > > > > but
> > > > > > > only if there are other, more specific guarantees in addition.
> This
> > > > > seems
> > > > > > > to cede the point that a contract between readers and writers
> makes
> > > > > > sense,
> > > > > > > but only if it is a _specific_ contract. But the proposed
> additions
> > > > > don't
> > > > > > > make sense in the same way.
> > > > > > >
> > > > > > > Take, for example, the suggestion to also disallow zero-length
> > > > vectors.
> > > > > > At
> > > > > > > a glance this seems similar, but I think it assumes a specific
> > > > context
> > > > > > and
> > > > > > > is better suited to be a constraint for some use cases rather
> than
> > > > part
> > > > > > of
> > > > > > > what is considered valid for the type. That's because
> zero-length
> > > > > vectors
> > > > > > > are valid vectors with a length. You can calculate Euclidean or
> > > > > Manhattan
> > > > > > > distance and use them in many contexts. They don't work if
> you're
> > > > > > > normalizing to a length of 1, but it's a choice to normalize
> and
> > > use
> > > > a
> > > > > > dot
> > > > > > > product to calculate cosine distance. And at this point, we're
> > > > talking
> > > > > > > about the usefulness of restrictions for the type, not a
> Platonic
> > > > > ideal.
> > > > > > >
> > > > > > > I think the restriction that makes the most sense is that
> vectors
> > > > have
> > > > > a
> > > > > > > length and can be compared in distance calculations. 0-length
> > > vectors
> > > > > are
> > > > > > > valid because they point somewhere and have a length. We
> shouldn't
> > > > > > restrict
> > > > > > > uses to only cosine distance over normalized vectors. Those
> are use
> > > > > case
> > > > > > > dependent choices and there are other valid ones. I also don't
> > > think
> > > > > that
> > > > > > > the model used to produce a vector is part of the vector type,
> just
> > > > > like
> > > > > > > the method used to produce a metric doesn't define its type:
> cosine
> > > > > > > similarity produces values between -1 and 1, but we store it
> in a
> > > > > > `float`,
> > > > > > > not a `float(cosine)`.
> > > > > > >
> > > > > > > Why does it make sense to have a finiteness restriction as
> part of
> > > > the
> > > > > > > type? Because that's what a (mathematical) vector is. A simple
> > > search
> > > > > > gives
> > > > > > > a pretty common definition: "A vector is an object that has
> both
> > > > > > magnitude
> > > > > > > and a direction". If a dimension/element is NaN, null, or
> infinite,
> > > > the
> > > > > > > vector doesn't point anywhere and it doesn't have a magnitude.
> Not
> > > > only
> > > > > > can
> > > > > > > it not be used, it doesn't meet the most basic definition of
> the
> > > > type.
> > > > > > >
> > > > > > > Ryan
> > > > > > >
> > > > > > > On Thu, Oct 15, 2026 at 1:07 AM Jan Finis <jp...@gmail.com>
> > > wrote:
> > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > Daniel,
> > > > > > > >
> > > > > > > > alright, so no physical optimizations intended.  We only want
> > > this
> > > > > for
> > > > > > > > semantic reasons, not for any significant physical
> optimization.
> > > If
> > > > > > we're
> > > > > > > > all on the same page here, that is good 👍.
> > > > > > > >
> > > > > > > > To phrase my remaining problem with this more concretely:
> > > > > > > > *Let's say we add this type. What cool new stuff can we now
> do
> > > that
> > > > > we
> > > > > > > > couldn't do before?*
> > > > > > > > That is the litmus test that a type has to pass to be useful
> > > IMHO.
> > > > > And
> > > > > > > here
> > > > > > > > I can't come up with a tangible use case that wasn't possible
> > > > before.
> > > > > > > > Especially if we one day have fixed size lists in Parquet,
> the
> > > > > > remaining
> > > > > > > > niche for this type becomes so small, that the added
> complexity
> > > > isn't
> > > > > > > worth
> > > > > > > > the benefit IMHO.
> > > > > > > >
> > > > > > > > As I guess most people on here are active maintainers of a
> > > Parquet
> > > > > > > > implementation: Concretely, what would you be using this
> type for
> > > > if
> > > > > we
> > > > > > > had
> > > > > > > > it? Would you allow math only on fields tagged with this
> type?
> > > What
> > > > > if
> > > > > > > that
> > > > > > > > math yields INF or NaN? Not allowed?
> > > > > > > >
> > > > > > > > Especially if we also discuss having both VECTOR and
> > > FINITE_VECTOR.
> > > > > How
> > > > > > > > will you guys wire this up with the type systems of your
> engines?
> > > > Do
> > > > > > your
> > > > > > > > engines also have two types here? Do you type-decay a
> > > FINITE_VECTOR
> > > > > to
> > > > > > a
> > > > > > > > VECTOR once someone does a multiplication that could yield
> INF?
> > > By
> > > > > > > itself,
> > > > > > > > a FINITE_VECTOR sounds good. Holistically looked at from a
> type
> > > > > system
> > > > > > > > perspective, it comes with a lot of complexity. Do the
> benefits
> > > > > really
> > > > > > > > outweigh the complexity here? That's why I would rather have
> the
> > > > bare
> > > > > > > > minimum: A fixed size array (list) of floats or at least
> just a
> > > > > vector
> > > > > > > > without finite. Then all these detailed questions go away.
> > > > > > > >
> > > > > > > > I'll take this one step further... Why is there any debate
> about
> > > > this
> > > > > > at
> > > > > > > > > all? Why can't a type simply be added to the
> *specification* as
> > > > an
> > > > > > > > > administrative detail and be done?
> > > > > > > >
> > > > > > > >
> > > > > > > > Andrew,
> > > > > > > >
> > > > > > > > Parquet is supposed to be a lingua franca of data. The
> > > > specification
> > > > > is
> > > > > > > the
> > > > > > > > authoritative truth for this language and all
> implementations are
> > > > > > > expected
> > > > > > > > to implement as much as possible from this spec, so anything
> in
> > > the
> > > > > > spec
> > > > > > > > does causes implementation effort. So just adding anything to
> > > this
> > > > > spec
> > > > > > > > without debate is not a good idea. It is even the other way
> > > around:
> > > > > We
> > > > > > > need
> > > > > > > > to have way higher rigor with the spec than we need to have
> with
> > > > any
> > > > > > > > implementations. Implementations can be changed, the spec
> needs
> > > to
> > > > be
> > > > > > > > stable.
> > > > > > > >
> > > > > > > > A tag-only type that doesn't come with any restrictions can
> be
> > > > added
> > > > > > with
> > > > > > > > little effort. But here we want to tie restrictions to it (no
> > > > > NaN/Inf),
> > > > > > > so
> > > > > > > > we would expect implementations to enforce these. So there is
> > > some
> > > > > > > > implementation effort expected here.
> > > > > > > >
> > > > > > > > Also, even if we only wanted a tag type, I would still argue
> that
> > > > you
> > > > > > > > shouldn't add those without a debate. What if A wants XML, B
> > > wants
> > > > > > YAML,
> > > > > > > C
> > > > > > > > wants quaternions, D wants address, E wants phone number. If
> we
> > > > just
> > > > > > > added
> > > > > > > > any types that are "just tags", Parquet logical types would
> just
> > > > > > become a
> > > > > > > > trash dump of any type that anyone ever thought to be
> useful. So
> > > > > > rigorous
> > > > > > > > assessment of each new type candidate is necessary.
> > > > > > > >
> > > > > > > > Finally, many engines couple parquet with another type system
> > > > (e.g.,
> > > > > > > Arrow
> > > > > > > > or SQL). So even with a tag only type, you at least still
> have
> > > the
> > > > > > > > implementation burden of wiring this up with your engine's
> type
> > > > > system.
> > > > > > > >
> > > > > > > > So, just because logical types are "just" logical, that
> doesn't
> > > > mean
> > > > > we
> > > > > > > can
> > > > > > > > add them without any rigor.
> > > > > > > >
> > > > > > > >
> > > > > > > > A few naive questions:
> > > > > > > > > 1. Would a physical definition with NANs & Inf+/- allow
> > > efficient
> > > > > > > > > encodings like ALP?
> > > > > > > > > 2. If the physical representation allows values that make
> > > > encodings
> > > > > > > like
> > > > > > > > > ALP impossible, can a tighter Logical type allow different
> > > > > encodings?
> > > > > > > > > 3. Can we RLE or binpack a vector?
> > > > > > > >
> > > > > > > >
> > > > > > > > The simple general answer here is that all these things do
> not
> > > > > require
> > > > > > a
> > > > > > > > logical type. Writers can already do any such optimization
> if the
> > > > > > actual
> > > > > > > > data contains no NaNs & Infs. All these physical
> optimizations
> > > just
> > > > > > > require
> > > > > > > > no actual NaNs/Infs to be present in a page, they don't need
> a
> > > > > > guarantee
> > > > > > > > that the whole file won't contain any.
> > > > > > > >
> > > > > > > > Cheers,
> > > > > > > > Jan
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > Am Do., 15. Okt. 2026 um 04:28 Uhr schrieb <
> > > > kurtiscwright@gmail.com
> > > > > >:
> > > > > > > >
> > > > > > > > > Hi Everyone,
> > > > > > > > >
> > > > > > > > > Glad to see a Vector type being added. I have a few use
> cases I
> > > > > want
> > > > > > to
> > > > > > > > > pursue once we have an agreed upon definition.
> > > > > > > > >
> > > > > > > > > In regards to that definition discussion. I feel
> comfortable
> > > with
> > > > > the
> > > > > > > > idea
> > > > > > > > > of the semantic meaning Dan raises though I can’t shake a
> > > slight
> > > > > > > > discomfort
> > > > > > > > > (maybe FOMO) with limiting values in the physical
> definition.
> > > > > > > > >
> > > > > > > > > A few naive questions:
> > > > > > > > > 1. Would a physical definition with NANs & Inf+/- allow
> > > efficient
> > > > > > > > > encodings like ALP?
> > > > > > > > > 2. If the physical representation allows values that make
> > > > encodings
> > > > > > > like
> > > > > > > > > ALP impossible, can a tighter Logical type allow different
> > > > > encodings?
> > > > > > > > > 3. Can we RLE or binpack a vector?
> > > > > > > > >
> > > > > > > > > Best Regards,
> > > > > > > > > Kurtis C. Wright
> > > > > > > > > > On Oct 14, 2026, at 12:45, Daniel Weeks <
> dweeks@apache.org>
> > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > Hey Jan,
> > > > > > > > > >
> > > > > > > > > > There's a lot that I disagree with respect to the
> framing of
> > > > > these
> > > > > > > > > > arguments and how we should be thinking about introducing
> > > > vector
> > > > > > as a
> > > > > > > > > > logical type.  There's a repeated tendency to let the
> > > physical
> > > > > > > > > > representation or performance drive discussion around
> types
> > > as
> > > > > > > opposed
> > > > > > > > to
> > > > > > > > > > what it represents.
> > > > > > > > > >
> > > > > > > > > > In parquet, establishing semantic meaning is the purpose
> of
> > > > > logical
> > > > > > > > > types.
> > > > > > > > > > The logical type separates how to interpret a value from
> its
> > > > > > physical
> > > > > > > > > > representation.  We can represent a timestamp with INT64
> or
> > > > even
> > > > > a
> > > > > > > > string
> > > > > > > > > > value, so why have a type for it? Even though a
> > > > FIXED_SIZED_LIST
> > > > > > and
> > > > > > > > > VECTOR
> > > > > > > > > > may have the exact same representation and performance,
> they
> > > > have
> > > > > > > > > different
> > > > > > > > > > semantic meaning.  Databases and engines across the
> ecosystem
> > > > > > already
> > > > > > > > > > recognize this differentiation and have separate
> > > > representations.
> > > > > > > The
> > > > > > > > > > choice to separate these isn't because the discussion is
> too
> > > > > > tedious;
> > > > > > > > > > they're fundamentally different things.
> > > > > > > > > >
> > > > > > > > > > The finite requirement is what makes a vector a
> reflection of
> > > > > what
> > > > > > it
> > > > > > > > > > represents.  As we've discussed in the thread, non-finite
> > > > values
> > > > > in
> > > > > > > the
> > > > > > > > > > representation violate the properties of a vector and the
> > > value
> > > > > > > ceases
> > > > > > > > to
> > > > > > > > > > be a vector.  On the other hand, a 0 vector is still a
> vector
> > > > > (you
> > > > > > > can
> > > > > > > > > > argue the utility of it, but it is a valid vector).
> > > > > > Implementations
> > > > > > > > can
> > > > > > > > > > choose what they validate or how to handle execution
> errors;
> > > > > that's
> > > > > > > an
> > > > > > > > > > implementation concern, not a requirement.  Operations
> > > > performed
> > > > > > on a
> > > > > > > > > > vector depend on the context in which they are used, but
> the
> > > > > > > > > representation
> > > > > > > > > > doesn't need to solve scenarios where that application
> > > doesn't
> > > > > make
> > > > > > > > > sense.
> > > > > > > > > >
> > > > > > > > > > The guarantees or context needed to use a vector do not
> need
> > > to
> > > > > be
> > > > > > > > > inherent
> > > > > > > > > > in the type for it to be a useful type.  We don't need
> the
> > > > > currency
> > > > > > > > used
> > > > > > > > > > for decimal or whether a floating point is a normalized
> value
> > > > as
> > > > > > part
> > > > > > > > of
> > > > > > > > > > the type to make use of those types.  This conflates the
> use
> > > > > cases
> > > > > > > with
> > > > > > > > > the
> > > > > > > > > > type system and unnecessarily attempts to draw the
> context
> > > into
> > > > > the
> > > > > > > > type
> > > > > > > > > > definition.  Similarly, you don't need to capture
> > > > normalization,
> > > > > > > model,
> > > > > > > > > > etc. as part of the type for it to be useful.
> > > > > > > > > >
> > > > > > > > > > So the reason we want to add a vector type (or
> FINITE_VECTOR)
> > > > is
> > > > > > > > because
> > > > > > > > > we
> > > > > > > > > > want to store vectors with the semantic meaning and
> > > > requirements
> > > > > > > based
> > > > > > > > on
> > > > > > > > > > what they represent.
> > > > > > > > > >
> > > > > > > > > > -Dan
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >> On Wed, Oct 14, 2026 at 1:22 AM Jan Finis <
> > > jpfinis@gmail.com>
> > > > > > > wrote:
> > > > > > > > > >>
> > > > > > > > > >> A bit late to the party, but here we go:
> > > > > > > > > >>
> > > > > > > > > >> I have the strong sentiment that the FINITE_VECTOR type
> as
> > > it
> > > > is
> > > > > > > > > currently
> > > > > > > > > >> being discussed will not be very useful and just
> increase
> > > > > > > > implementation
> > > > > > > > > >> complexity. The arguments for the type are all quite
> weak
> > > > > (details
> > > > > > > see
> > > > > > > > > >> below). The FINITE part doesn't seem to yield any value
> and
> > > is
> > > > > > not a
> > > > > > > > > useful
> > > > > > > > > >> restriction by itself. The vector (i.e. fixed length)
> part
> > > > > should
> > > > > > > > > rather be
> > > > > > > > > >> generalized instead of making a one-off thing. In
> general,
> > > the
> > > > > > > > existing
> > > > > > > > > >> types are already working well for uncompressed
> vectors. The
> > > > > real
> > > > > > > > > challenge
> > > > > > > > > >> that would bring real benefits is supporting
> quantization.
> > > > This
> > > > > is
> > > > > > > > where
> > > > > > > > > >> you get orders of magnitude improvements to storage
> space
> > > and
> > > > > > > > processing
> > > > > > > > > >> speed.
> > > > > > > > > >>
> > > > > > > > > >> Let's first discuss the FINITE part. So far, the only
> > > > arguments
> > > > > I
> > > > > > > have
> > > > > > > > > read
> > > > > > > > > >> for this are:
> > > > > > > > > >>
> > > > > > > > > >> 1. many other vector databases have it
> > > > > > > > > >> 2. vectors are the hot stuff right now, so Parquet
> better
> > > > boast
> > > > > > > having
> > > > > > > > > >> "native" type support for vectors
> > > > > > > > > >> 3. by focussing on this common special case, we can make
> > > rapid
> > > > > > > > progress,
> > > > > > > > > >> while a general solution would take too long to specify
> > > > > > > > > >> 4. to give a guarantee to readers that no INF/NaN/NULL
> will
> > > be
> > > > > in
> > > > > > > the
> > > > > > > > > >> vector
> > > > > > > > > >>
> > > > > > > > > >> the first two arguments are quite weak by themselves.
> Vector
> > > > > > > databases
> > > > > > > > > are
> > > > > > > > > >> very specialized engines. Such engines can of course be
> very
> > > > > > > > > opinionated on
> > > > > > > > > >> their data types. A general format like Parquet must
> instead
> > > > > weigh
> > > > > > > > > whether
> > > > > > > > > >> a special type is worth the added complexity over a more
> > > > > general,
> > > > > > > > widely
> > > > > > > > > >> applicable type. In this case, it is IMHO not at all.
> And
> > > just
> > > > > > > having
> > > > > > > > a
> > > > > > > > > >> type called "vector" for advertisement reasons to make
> > > Parquet
> > > > > > > appear
> > > > > > > > > more
> > > > > > > > > >> modern seems a non-goal in the first place.
> > > > > > > > > >>
> > > > > > > > > >> The third "time to market" argument is valid for systems
> > > that
> > > > > > don't
> > > > > > > > > need a
> > > > > > > > > >> stable interface. It is a very dangerous attitude for
> > > formats
> > > > > like
> > > > > > > > > Parquet
> > > > > > > > > >> that must be stable long term and should have a
> > > consolidated,
> > > > > > > concise,
> > > > > > > > > >> non-redundant type system. For such long term storage
> > > formats,
> > > > > we
> > > > > > > need
> > > > > > > > > to
> > > > > > > > > >> take the time to build the right thing, as any
> shortcuts we
> > > > take
> > > > > > > will
> > > > > > > > > stay
> > > > > > > > > >> as blemishes in the format forever. If I read the
> discussion
> > > > > > > > correctly,
> > > > > > > > > >> this is actually the main argument as of now: We have
> found
> > > > that
> > > > > > it
> > > > > > > is
> > > > > > > > > hard
> > > > > > > > > >> to align on how a fixed size list type should look
> like, so
> > > we
> > > > > > want
> > > > > > > to
> > > > > > > > > go
> > > > > > > > > >> the easier route of a specialized type. This seems like
> a
> > > > > > > > > >> dangerous shortcut fueled by feature recency bias. Ask
> > > > yourself:
> > > > > > > Would
> > > > > > > > > we
> > > > > > > > > >> want this, if Parquet already had a fixed size list
> type?
> > > > > > > > > >>
> > > > > > > > > >> So finally, the only technical argument is the fourth
> one:
> > > "It
> > > > > > > signals
> > > > > > > > > >> semantics and ensures that no bad data lands in the
> > > format". I
> > > > > see
> > > > > > > two
> > > > > > > > > big
> > > > > > > > > >> problems with this:
> > > > > > > > > >>
> > > > > > > > > >> 1. Readers *still* need to test for NaN / INF because
> > > Parquet
> > > > > > files
> > > > > > > > may
> > > > > > > > > be
> > > > > > > > > >> corrupt. Nothing can stop a malicious or buggy writer
> from
> > > > > > > physically
> > > > > > > > > >> writing NaN or INF. *So you save no single line of code
> in a
> > > > > > reader.
> > > > > > > > You
> > > > > > > > > >> save no branches, as you still need to handle the
> INF/NaN
> > > > case.*
> > > > > > So
> > > > > > > > this
> > > > > > > > > >> doesn't help with performance at all. The only
> difference
> > > is,
> > > > > that
> > > > > > > the
> > > > > > > > > >> error message changes from "found an invalid INF value
> in
> > > the
> > > > > > data"
> > > > > > > to
> > > > > > > > > >> "Your parquet file seems to be corrupt". It is actually
> > > quite
> > > > > the
> > > > > > > > > opposite,
> > > > > > > > > >> readers and writers get more complex having to handle
> this
> > > new
> > > > > > type.
> > > > > > > > > >>
> > > > > > > > > >> If you now argue that it's not about performance, it's
> about
> > > > > > defined
> > > > > > > > > >> semantics and erroring out early in writers, then the
> second
> > > > > > problem
> > > > > > > > > >> arises:
> > > > > > > > > >>
> > > > > > > > > >> 2. The proposed no-NaN/INF/NULL guarantee is just one of
> > > > > multiple
> > > > > > > > > >> guarantees that you need for sensible vector
> processing. If
> > > > "no
> > > > > > > > > >> INF/NaN/NULL" was the only guarantee you needed, I would
> > > agree
> > > > > > that
> > > > > > > > > >> FINITE_VECTOR is useful, but it is by no means the only
> > > > > guarantee
> > > > > > > you
> > > > > > > > > need,
> > > > > > > > > >> instead, depending on the distance metric, you also
> need the
> > > > > > > following
> > > > > > > > > >> guarantees. E.g.:
> > > > > > > > > >>
> > > > > > > > > >>   - No vector may be the 0 vector, otherwise, e.g.,
> cosine
> > > > > > > similarity
> > > > > > > > is
> > > > > > > > > >>   undefined. Cosine similarity is often used as
> default, so
> > > > this
> > > > > > is
> > > > > > > > > >> actually
> > > > > > > > > >>   quite important in practice.
> > > > > > > > > >>   - If we want to use a dot product, then all vectors
> need
> > > to
> > > > > > > > guarantee
> > > > > > > > > a
> > > > > > > > > >>   length of 1. This would actually be quite useful for
> > > > > performance
> > > > > > > > > >> reasons.
> > > > > > > > > >>   - The vectors must come from the same model as the
> vector
> > > > used
> > > > > > for
> > > > > > > > the
> > > > > > > > > >>   search. Otherwise, the search yields a numeric
> result, but
> > > > > that
> > > > > > > > result
> > > > > > > > > >> has
> > > > > > > > > >>   no sensible meaning.
> > > > > > > > > >>
> > > > > > > > > >> So, what use is a type that just guarantees part of the
> > > things
> > > > > you
> > > > > > > > need
> > > > > > > > > for
> > > > > > > > > >> vector search but not others? You still need additional
> > > > > guarantees
> > > > > > > in
> > > > > > > > > >> addition to that type. What makes INF/NaN so important
> in
> > > > > > comparison
> > > > > > > > to,
> > > > > > > > > >> say, "no 0 vector present", that it warrants an own
> type,
> > > > while
> > > > > > the
> > > > > > > > > other
> > > > > > > > > >> guarantees do not? If we really want this type as a
> basis
> > > for
> > > > > > > > > cross-engine
> > > > > > > > > >> sharing of vectors, then it needs to give *all*
> guarantees.
> > > > > > > > > >>
> > > > > > > > > >> I'm not against a specialized vector type if it really
> > > carries
> > > > > > *all*
> > > > > > > > > >> information to be useful by itself. If it carried all
> that
> > > > > > > > information,
> > > > > > > > > >> then a reading engine could indeed validate that only
> > > > meaningful
> > > > > > > > vector
> > > > > > > > > >> searches are executed. But as long as it is just a
> > > half-baked
> > > > > > thing,
> > > > > > > > it
> > > > > > > > > is
> > > > > > > > > >> not useful by itself and just a burden on each
> implementer
> > > and
> > > > > > > > > maintainer.
> > > > > > > > > >> If we need extra information anyway, why not just stick
> to a
> > > > > > general
> > > > > > > > > type
> > > > > > > > > >> like VECTOR (without FINITE).
> > > > > > > > > >>
> > > > > > > > > >> If you're a proponent of FINITE_VECTOR, ask yourself (or
> > > tell
> > > > > us)
> > > > > > > what
> > > > > > > > > >> advantage you will *really* get over just a (potentially
> > > fixed
> > > > > > size)
> > > > > > > > > List
> > > > > > > > > >> of Float? As discussed, it's not performance. It's also
> not
> > > a
> > > > > > > semantic
> > > > > > > > > >> marker that is sufficient by itself. It's just a small
> hint
> > > > that
> > > > > > > > allows
> > > > > > > > > >> writers to error out once they see a NaN or INF in the
> > > input,
> > > > > > while
> > > > > > > > they
> > > > > > > > > >> will still happily write a bogus 0 vector. This seems
> to be
> > > of
> > > > > > zero
> > > > > > > > > >> benefit.
> > > > > > > > > >>
> > > > > > > > > >> Anecdotally, we currently handle petabytes of vectors in
> > > > > > production
> > > > > > > in
> > > > > > > > > >> Data360. We simply store them in a List of Float, with
> an
> > > > > > optimized
> > > > > > > > > R-Level
> > > > > > > > > >> reader implementation that rapidly checks that all
> vectors
> > > are
> > > > > the
> > > > > > > > same
> > > > > > > > > >> length. This solution works just fine. The overhead of
> > > R-level
> > > > > > > > decoding
> > > > > > > > > is
> > > > > > > > > >> barely measurable. The check is super fast and R-levels
> are
> > > > tiny
> > > > > > in
> > > > > > > > > >> comparison to the payload vectors, so also the added
> I/O is
> > > > > > > > negligible.
> > > > > > > > > >> This is also why I'm so firm in my sentiment: We are
> > > running a
> > > > > > > > > production
> > > > > > > > > >> grade system processing trillions of vectors right now
> and
> > > see
> > > > > no
> > > > > > > big
> > > > > > > > > >> issues with the types that Parquet currently provides.
> > > > > > > > > >>
> > > > > > > > > >> As I don't just want to be a naysayer and point out
> problems
> > > > > with
> > > > > > > the
> > > > > > > > > >> current design, here are the directions that I would
> propose
> > > > > > indeed:
> > > > > > > > > >>
> > > > > > > > > >>   - a general fixed size list type. Don't do shortcuts
> just
> > > > > > because
> > > > > > > > the
> > > > > > > > > >>   discussion is too tedious.
> > > > > > > > > >>   - a vector type that indeed carries enough
> information to
> > > be
> > > > > > > really
> > > > > > > > > >>   useful for vector search by itself. (e.g., model name
> /
> > > > > version,
> > > > > > > > > >> optional
> > > > > > > > > >>   guarantees for no 0 vector / normalized vector
> length).
> > > With
> > > > > > such
> > > > > > > a
> > > > > > > > > >> type,
> > > > > > > > > >>   you could really do well defined vector searches even
> if
> > > the
> > > > > > > writing
> > > > > > > > > >> engine
> > > > > > > > > >>   is completely different to the reading one.
> > > > > > > > > >>   - Standardize quantizations. This will be a real game
> > > > changer
> > > > > > for
> > > > > > > > > >>   performance.
> > > > > > > > > >>
> > > > > > > > > >> Cheers,
> > > > > > > > > >> Jan
> > > > > > > > > >>
> > > > > > > > > >>
> > > > > > > > > >>
> > > > > > > > > >>
> > > > > > > > > >>
> > > > > > > > > >>
> > > > > > > > > >>> Am Mi., 14. Okt. 2026 um 03:36 Uhr schrieb Gang Wu <
> > > > > > > ustcwg@gmail.com
> > > > > > > > >:
> > > > > > > > > >>>
> > > > > > > > > >>> I think separating them into FINITE_VECTOR and
> > > > > FIXED_LENGTH_LIST
> > > > > > > is a
> > > > > > > > > >> good
> > > > > > > > > >>> idea, though I think renaming FINITE_VECTOR to VECTOR
> looks
> > > > > more
> > > > > > > > > natural.
> > > > > > > > > >>> Should we move forward with proposing
> FINITE_VECTOR/VECTOR
> > > > > first?
> > > > > > > > > >>>
> > > > > > > > > >>> On Tue, Oct 13, 2026 at 6:26 PM Alkis Evlogimenos via
> dev <
> > > > > > > > > >>> dev@parquet.apache.org> wrote:
> > > > > > > > > >>>
> > > > > > > > > >>>> I suggest we go with VECTOR for the finite vector to
> keep
> > > > the
> > > > > > name
> > > > > > > > the
> > > > > > > > > >>> same
> > > > > > > > > >>>> as other engines/type systems. If we want a
> non-finite one
> > > > in
> > > > > > the
> > > > > > > > > >> future
> > > > > > > > > >>> we
> > > > > > > > > >>>> can name that something else or we add fixed size list
> > > that
> > > > > will
> > > > > > > > serve
> > > > > > > > > >>> the
> > > > > > > > > >>>> same purpose without a special name.
> > > > > > > > > >>>>
> > > > > > > > > >>>>
> > > > > > > > > >>>> On Thu, Oct 8, 2026 at 10:20 PM Ryan Blue <
> > > blue@apache.org>
> > > > > > > wrote:
> > > > > > > > > >>>>
> > > > > > > > > >>>>> A logical type would:
> > > > > > > > > >>>>> - Describe data for users and readers to understand
> what
> > > > the
> > > > > > file
> > > > > > > > > >>>> contains
> > > > > > > > > >>>>> - Signal that repetition and definition levels will
> > > > reflect a
> > > > > > > fixed
> > > > > > > > > >>>> number
> > > > > > > > > >>>>> of elements, which implementations can use to
> optimize
> > > > > decoding
> > > > > > > > > >>>>> - Guarantee that there are no null elements (because
> the
> > > > type
> > > > > > > > cannot
> > > > > > > > > >> be
> > > > > > > > > >>>>> applied to a list with optional elements)
> > > > > > > > > >>>>> - [FINITE_VECTOR] Guarantee that no elements are NaN
> or
> > > > > > Infinity
> > > > > > > > > >> values
> > > > > > > > > >>>> so
> > > > > > > > > >>>>> that the vectors have a finite length and are
> suitable
> > > for
> > > > > > > distance
> > > > > > > > > >>>>> calculations
> > > > > > > > > >>>>>
> > > > > > > > > >>>>> The finite value restriction is a restriction on
> writers
> > > to
> > > > > > > prevent
> > > > > > > > > >> bad
> > > > > > > > > >>>>> data from being written.
> > > > > > > > > >>>>>
> > > > > > > > > >>>>> Ignoring logical types that are not understood is a
> rule
> > > > for
> > > > > > > > > >> readers. A
> > > > > > > > > >>>>> reader can handle vectors and pass it on to another
> > > layer.
> > > > > For
> > > > > > > > > >>> instance,
> > > > > > > > > >>>> a
> > > > > > > > > >>>>> Parquet reader that doesn't support geometry passes
> > > values
> > > > as
> > > > > > > > > >> `binary`.
> > > > > > > > > >>>>> That binary value can be passed to a geometry UDF,
> even
> > > if
> > > > > the
> > > > > > > > > >>> underlying
> > > > > > > > > >>>>> engine and Parquet reader doesn't have support.
> > > > > > > > > >>>>>
> > > > > > > > > >>>>> A tool that doesn't understand FINITE_VECTOR can
> > > similarly
> > > > > > pass a
> > > > > > > > > >> list
> > > > > > > > > >>> of
> > > > > > > > > >>>>> floats along to a UDF. But it is still important to
> the
> > > > user
> > > > > > that
> > > > > > > > bad
> > > > > > > > > >>>>> values are not present in the data. In your example,
> bad
> > > > > values
> > > > > > > are
> > > > > > > > > >>>> vectors
> > > > > > > > > >>>>> that cause the centroid calculation to fail, like
> vectors
> > > > > with
> > > > > > > > > >> infinite
> > > > > > > > > >>>>> length.
> > > > > > > > > >>>>>
> > > > > > > > > >>>>> Ryan
> > > > > > > > > >>>>>
> > > > > > > > > >>>>> On Thu, Oct 8, 2026 at 12:54 PM Will Edwards via dev
> <
> > > > > > > > > >>>>> dev@parquet.apache.org>
> > > > > > > > > >>>>> wrote:
> > > > > > > > > >>>>>
> > > > > > > > > >>>>>> Could we clarify the use cases for a VECTOR
> annotation
> > > on
> > > > > > > > > >>> LIST<FLOAT>?
> > > > > > > > > >>>>>> Specifically, what does it make possible that's
> > > impossible
> > > > > > > today,
> > > > > > > > > >> and
> > > > > > > > > >>>>> what
> > > > > > > > > >>>>>> does it make easier that's currently hard?
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>> I'm struggling to find a strong answer. I'd like
> every
> > > > > engine
> > > > > > to
> > > > > > > > > >>> offer
> > > > > > > > > >>>>>> vector functions on lists of floats without needing
> a
> > > new
> > > > > > > logical
> > > > > > > > > >>> type.
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>> To be clear, I'm still enthusiastic about a vector
> type
> > > > that
> > > > > > > > > >>> describes
> > > > > > > > > >>>>> how
> > > > > > > > > >>>>>> to unpack and scale composite representations
> (quantised
> > > > > codes
> > > > > > > > plus
> > > > > > > > > >>>>> scales,
> > > > > > > > > >>>>>> codebooks, etc.), in the direction ML is going. That
> > > adds
> > > > > > > > > >> information
> > > > > > > > > >>>>>> engines don't have. A label on LIST<FLOAT> doesn't:
> > > > engines
> > > > > > > > already
> > > > > > > > > >>>>>> understand the physical type, and many already offer
> > > > vector
> > > > > > > > > >> functions
> > > > > > > > > >>>> on
> > > > > > > > > >>>>>> it. I don't see a performance benefit, and
> NaN/null/Inf
> > > > > checks
> > > > > > > are
> > > > > > > > > >>>> cheap
> > > > > > > > > >>>>>> enough that engines should do them anyway. The label
> > > also
> > > > > > > doesn't
> > > > > > > > > >> say
> > > > > > > > > >>>>> what
> > > > > > > > > >>>>>> you can do with the column. Knowing that a column
> holds
> > > > > > > embeddings
> > > > > > > > > >>>>> doesn't
> > > > > > > > > >>>>>> say how they can be used, just as zipcode and age
> are
> > > both
> > > > > > > > integers
> > > > > > > > > >>> but
> > > > > > > > > >>>>>> multiplying them is meaningless.
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>> My concern is the opposite effect. We're
> establishing
> > > that
> > > > > > > engines
> > > > > > > > > >>>> should
> > > > > > > > > >>>>>> ignore logical types they don't understand. So a
> tool
> > > that
> > > > > > > doesn't
> > > > > > > > > >>> know
> > > > > > > > > >>>>>> VECTOR will read the column as LIST<FLOAT>, let the
> user
> > > > > > compute
> > > > > > > > > >>>>> centroids
> > > > > > > > > >>>>>> (say), and write plain LIST<FLOAT> back out. If
> other
> > > > > engines
> > > > > > > > > >>> restrict
> > > > > > > > > >>>>>> embedding functions to VECTOR-annotated columns,
> that
> > > file
> > > > > > stops
> > > > > > > > > >>>> working
> > > > > > > > > >>>>>> with them, through no fault of the user. An
> annotation
> > > > that
> > > > > > > > doesn't
> > > > > > > > > >>>>> change
> > > > > > > > > >>>>>> what the data is shouldn't be able to make it
> unusable.
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>> Am I missing a use case?
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>> Best, Will
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>> On Thu, 8 Oct 2026 at 19:53, Steven Wu <
> > > > > stevenz3wu@gmail.com>
> > > > > > > > > >> wrote:
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>>> I also think a FINITE_VECTOR type brings a lot of
> > > values
> > > > to
> > > > > > > many
> > > > > > > > > >>> use
> > > > > > > > > >>>>>> cases.
> > > > > > > > > >>>>>>> It is would be great to make progress on the
> > > > FINITE_VECTOR,
> > > > > > and
> > > > > > > > > >> not
> > > > > > > > > >>>> to
> > > > > > > > > >>>>>> fit
> > > > > > > > > >>>>>>> every use cases into the FIXED_LENGTH_LIST.
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> On Mon, Oct 5, 2026 at 1:26 PM Ryan Blue <
> > > > blue@apache.org>
> > > > > > > > > >> wrote:
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>>> Going through this thread again, it looks like
> there
> > > > are a
> > > > > > few
> > > > > > > > > >>>> things
> > > > > > > > > >>>>>>> that
> > > > > > > > > >>>>>>>> people generally agree on:
> > > > > > > > > >>>>>>>>
> > > > > > > > > >>>>>>>> - In the short term, we want to introduce a
> logical
> > > type
> > > > > or
> > > > > > > > > >>> types;
> > > > > > > > > >>>> a
> > > > > > > > > >>>>>>> better
> > > > > > > > > >>>>>>>> physical representation can be done later. We can
> > > focus
> > > > on
> > > > > > > > > >>> logical
> > > > > > > > > >>>>>> types.
> > > > > > > > > >>>>>>>> - There is precedent in logical types for
> restricting
> > > > > > values,
> > > > > > > > > >>> like
> > > > > > > > > >>>>> UTF8
> > > > > > > > > >>>>>>>> strings to contain only UTF-8 codepoints. I think
> we
> > > > also
> > > > > > > agree
> > > > > > > > > >>>> that
> > > > > > > > > >>>>> we
> > > > > > > > > >>>>>>>> could restrict vector elements to non-nested or
> > > > primitive
> > > > > > > > > >> types.
> > > > > > > > > >>>>>>>> - There are many examples of databases that
> restrict
> > > > their
> > > > > > > > > >> vector
> > > > > > > > > >>>>> types
> > > > > > > > > >>>>>>> to
> > > > > > > > > >>>>>>>> non-null, finite elements.
> > > > > > > > > >>>>>>>> - Some people want to have write-time guarantees
> that
> > > > > > vectors
> > > > > > > > > >>> have
> > > > > > > > > >>>>> only
> > > > > > > > > >>>>>>>> non-null, finite elements
> > > > > > > > > >>>>>>>> - Some people want to extend this work to include
> > > > tensors,
> > > > > > > > > >>> shorter
> > > > > > > > > >>>>>>> floating
> > > > > > > > > >>>>>>>> point representations, variable lengths, NaN or
> > > Infinity
> > > > > > > > > >>> elements,
> > > > > > > > > >>>>> etc.
> > > > > > > > > >>>>>>>>
> > > > > > > > > >>>>>>>> It sounds like we are nearly there, but have two
> > > halves
> > > > of
> > > > > > the
> > > > > > > > > >>>>>> community
> > > > > > > > > >>>>>>>> that care about different things. That just means
> we
> > > > > should
> > > > > > > > > >> have
> > > > > > > > > >>>>>> multiple
> > > > > > > > > >>>>>>>> types. It's entirely reasonable to want a type
> that
> > > > allows
> > > > > > NaN
> > > > > > > > > >>>>> values,
> > > > > > > > > >>>>>>> and
> > > > > > > > > >>>>>>>> it's also reasonable for those of us building
> > > databases,
> > > > > > table
> > > > > > > > > >>>>>> standards,
> > > > > > > > > >>>>>>>> and vector indexes to care about guarantees that
> > > > elements
> > > > > > are
> > > > > > > > > >>>> finite.
> > > > > > > > > >>>>>>>> There's more than enough examples to show that
> this
> > > is a
> > > > > > > > > >>> legitimate
> > > > > > > > > >>>>> use
> > > > > > > > > >>>>>>>> case.
> > > > > > > > > >>>>>>>>
> > > > > > > > > >>>>>>>> Since I don't see the concerns of the different
> use
> > > > cases
> > > > > > > > > >>> changing,
> > > > > > > > > >>>>> it
> > > > > > > > > >>>>>>>> looks like we need to find a compromise to move
> > > > forward. I
> > > > > > > > > >>> propose
> > > > > > > > > >>>>> that
> > > > > > > > > >>>>>>> we
> > > > > > > > > >>>>>>>> introduce a FINITE_VECTOR type and separately a
> > > > > > > > > >> FIXED_LENGTH_LIST
> > > > > > > > > >>>> or
> > > > > > > > > >>>>>>>> another VECTOR type that allows infinite and NaN
> > > values.
> > > > > > > > > >>>>>>>>
> > > > > > > > > >>>>>>>> Can we agree on a compromise? Is there a strong
> reason
> > > > to
> > > > > > try
> > > > > > > > > >> to
> > > > > > > > > >>>> fit
> > > > > > > > > >>>>>>> these
> > > > > > > > > >>>>>>>> use cases into a single type? What if that means
> not
> > > > > making
> > > > > > > > > >>>> progress?
> > > > > > > > > >>>>>>>>
> > > > > > > > > >>>>>>>> Ryan
> > > > > > > > > >>>>>>>>
> > > > > > > > > >>>>>>>> On Mon, Oct 5, 2026 at 12:15 PM Philipp Fischbeck
> via
> > > > dev
> > > > > <
> > > > > > > > > >>>>>>>> dev@parquet.apache.org> wrote:
> > > > > > > > > >>>>>>>>
> > > > > > > > > >>>>>>>>> Thinking about this more, I am also fine with
> pulling
> > > > out
> > > > > > the
> > > > > > > > > >>>>>> “NaN/Inf”
> > > > > > > > > >>>>>>>>> requirement. Either into a type boolean (my
> > > > preference),
> > > > > or
> > > > > > > > > >>>> putting
> > > > > > > > > >>>>>> no
> > > > > > > > > >>>>>>>>> restrictions on the Parquet level, allowing all
> IEEE
> > > > 754
> > > > > > > > > >> float
> > > > > > > > > >>>>>> numbers.
> > > > > > > > > >>>>>>>> If
> > > > > > > > > >>>>>>>>> some top-level application wants to put
> additional
> > > > > > > > > >> constraints
> > > > > > > > > >>> on
> > > > > > > > > >>>>> the
> > > > > > > > > >>>>>>>> data
> > > > > > > > > >>>>>>>>> type, like Iceberg not allowing NaN/infinity, it
> can
> > > > > still
> > > > > > do
> > > > > > > > > >>> so
> > > > > > > > > >>>>> one
> > > > > > > > > >>>>>>>> layer
> > > > > > > > > >>>>>>>>> above. It seems like this was the biggest point
> of
> > > > > > discussion
> > > > > > > > > >>>>>>> remaining.
> > > > > > > > > >>>>>>>>> All these options work for me, and we should move
> > > > forward
> > > > > > > > > >> with
> > > > > > > > > >>>> this
> > > > > > > > > >>>>>>>>> proposal.
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>> Concerning additional element types like bf16,
> they
> > > can
> > > > > be
> > > > > > > > > >>>>> introduced
> > > > > > > > > >>>>>>> as
> > > > > > > > > >>>>>>>>> Parquet types (at least in the current LIST-based
> > > spec
> > > > > > > > > >>> proposal).
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>> Concerning Will’s suggestion around representing
> > > VECTOR
> > > > > as
> > > > > > a
> > > > > > > > > >>>>> variable
> > > > > > > > > >>>>>>>>> length list with a MIN/MAX_LENGTH metadata
> > > > restriction: I
> > > > > > > > > >> think
> > > > > > > > > >>>>> that
> > > > > > > > > >>>>>> a
> > > > > > > > > >>>>>>>>> dedicated logical VECTOR type still provides
> enough
> > > > > > benefits
> > > > > > > > > >> by
> > > > > > > > > >>>>>>>> describing
> > > > > > > > > >>>>>>>>> the semantics of the data, and allowing to
> capture
> > > > future
> > > > > > > > > >>> changes
> > > > > > > > > >>>>>>> around
> > > > > > > > > >>>>>>>>> statistics. In that case, the length is better
> > > captured
> > > > > as
> > > > > > an
> > > > > > > > > >>>> exact
> > > > > > > > > >>>>>>>> length
> > > > > > > > > >>>>>>>>> for this type specifically. I’m also not sure
> about
> > > > other
> > > > > > use
> > > > > > > > > >>>> cases
> > > > > > > > > >>>>>> for
> > > > > > > > > >>>>>>>>> MIN_LENGTH, so introducing a new type constraint
> > > system
> > > > > > just
> > > > > > > > > >>> for
> > > > > > > > > >>>>> the
> > > > > > > > > >>>>>>>>> MIN_LENGTH=MAX_LENGTH case for vectors doesn’t
> have
> > > > > enough
> > > > > > > > > >>> value
> > > > > > > > > >>>> in
> > > > > > > > > >>>>>> my
> > > > > > > > > >>>>>>>>> view.
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>> Best,
> > > > > > > > > >>>>>>>>> Philipp
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>> On Thu, Oct 1, 2026 at 9:00 AM Will Edwards via
> dev <
> > > > > > > > > >>>>>>>>> dev@parquet.apache.org>
> > > > > > > > > >>>>>>>>> wrote:
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>>> +1 Sebastian, a vector type suggests usage
> beyond
> > > > > > > > > >> normalized
> > > > > > > > > >>>>>>> embeddings
> > > > > > > > > >>>>>>>>> and
> > > > > > > > > >>>>>>>>>> engines really aren't going to be able to elide
> all
> > > > the
> > > > > > > > > >>> checks
> > > > > > > > > >>>>>>> anyway.
> > > > > > > > > >>>>>>>>>>
> > > > > > > > > >>>>>>>>>> Antoine, you suggested we explore metadata
> solutions
> > > > too
> > > > > > :)
> > > > > > > > > >>>>>>>>>>
> > > > > > > > > >>>>>>>>>> Today, a database supporting CHAR(N) or
> VARCHAR(n)
> > > > can't
> > > > > > > > > >>>> express
> > > > > > > > > >>>>>> this
> > > > > > > > > >>>>>>>> in
> > > > > > > > > >>>>>>>>>> Parquet.  Engines can write opaque blobs they
> > > > understand
> > > > > > in
> > > > > > > > > >>> the
> > > > > > > > > >>>>>>>> metadata
> > > > > > > > > >>>>>>>>>> (e.g. Arrow does) so the extra type info
> > > round-trips,
> > > > > but
> > > > > > > > > >>> this
> > > > > > > > > >>>>>> isn't
> > > > > > > > > >>>>>>>>>> standardized, portable, nor interoperable.
> > > > > > > > > >>>>>>>>>>
> > > > > > > > > >>>>>>>>>> So perhaps standardizing constraint metadata or
> new
> > > > > Thrift
> > > > > > > > > >>>> fields
> > > > > > > > > >>>>>> (my
> > > > > > > > > >>>>>>>>> pref)
> > > > > > > > > >>>>>>>>>> can generically support VECTOR(N) whilst also
> being
> > > > more
> > > > > > > > > >>>> broadly
> > > > > > > > > >>>>>>> useful
> > > > > > > > > >>>>>>>>> for
> > > > > > > > > >>>>>>>>>> other use cases and data types?
> > > > > > > > > >>>>>>>>>>
> > > > > > > > > >>>>>>>>>> Imagine metadata or Thrift fields to tag the
> > > > MIN_LENGTH
> > > > > > and
> > > > > > > > > >>>>>>> MAX_LENGTH
> > > > > > > > > >>>>>>>>> for
> > > > > > > > > >>>>>>>>>> Parquet's variable-length physcial types.  These
> > > would
> > > > > > > > > >> count
> > > > > > > > > >>>>>> logical
> > > > > > > > > >>>>>>>>> units
> > > > > > > > > >>>>>>>>>> (e.g. UTF8 code points for STRING) rather than
> > > bytes.
> > > > > > > > > >>>>>>>>>>
> > > > > > > > > >>>>>>>>>> Then one engine's VECTOR(N) can be written to
> > > Parquet
> > > > as
> > > > > > > > > >>>> LIST(N)
> > > > > > > > > >>>>>> with
> > > > > > > > > >>>>>>>>>> MIN_LENGTH = MAX_LENGTH = N and then readers -
> even
> > > > > > > > > >> different
> > > > > > > > > >>>>>>> engines -
> > > > > > > > > >>>>>>>>> can
> > > > > > > > > >>>>>>>>>> pick up these constraints if they support doing
> so.
> > > > We
> > > > > > > > > >> don't
> > > > > > > > > >>>>>>> actually
> > > > > > > > > >>>>>>>>> need
> > > > > > > > > >>>>>>>>>> a logical type to express a VECTOR then.
> > > > > > > > > >>>>>>>>>>
> > > > > > > > > >>>>>>>>>> This also allows standard SQL's CHAR(N) and
> > > VARCHAR(N)
> > > > > to
> > > > > > > > > >> be
> > > > > > > > > >>>>>>> expressed
> > > > > > > > > >>>>>>>> on
> > > > > > > > > >>>>>>>>>> UTF8 columns etc.
> > > > > > > > > >>>>>>>>>>
> > > > > > > > > >>>>>>>>>> This also seems generally useful in the future
> as we
> > > > > > > > > >> approach
> > > > > > > > > >>>>>>>> expressing
> > > > > > > > > >>>>>>>>>> other tensor data shapes and types.
> > > > > > > > > >>>>>>>>>>
> > > > > > > > > >>>>>>>>>> This won't change the mechanics: the storage
> type
> > > > > doesn't
> > > > > > > > > >>>> enforce
> > > > > > > > > >>>>>>> these
> > > > > > > > > >>>>>>>>>> constraints and engines must check them. It's
> > > > previously
> > > > > > > > > >>>>>> established
> > > > > > > > > >>>>>>>> that
> > > > > > > > > >>>>>>>>>> performance is just a software problem and
> there is
> > > no
> > > > > > > > > >>>>> fundamental
> > > > > > > > > >>>>>>>>>> performance advantage to the fixed size types in
> > > > > > > > > >> Dremel-like
> > > > > > > > > >>>>>> formats
> > > > > > > > > >>>>>>>> like
> > > > > > > > > >>>>>>>>>> Parquet.
> > > > > > > > > >>>>>>>>>>
> > > > > > > > > >>>>>>>>>> Best,
> > > > > > > > > >>>>>>>>>> Will
> > > > > > > > > >>>>>>>>>>
> > > > > > > > > >>>>>>>>>> On Wed, 30 Sept 2026 at 16:08, Sebastian
> Baunsgaard
> > > <
> > > > > > > > > >>>>>>>>> baunsgaard@apache.org
> > > > > > > > > >>>>>>>>>>>
> > > > > > > > > >>>>>>>>>> wrote:
> > > > > > > > > >>>>>>>>>>
> > > > > > > > > >>>>>>>>>>> After listening in on the last meeting
> Wednesday, I
> > > > > have
> > > > > > > > > >>> been
> > > > > > > > > >>>>>>>> thinking
> > > > > > > > > >>>>>>>>>>> about this quite a bit, and thought I would
> share
> > > my
> > > > 2
> > > > > > > > > >>> cents.
> > > > > > > > > >>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>> I suggest not requiring VECTOR elements to be
> > > finite,
> > > > > as
> > > > > > > > > >>>>> proposed
> > > > > > > > > >>>>>>> in
> > > > > > > > > >>>>>>>>> the
> > > > > > > > > >>>>>>>>>> PR
> > > > > > > > > >>>>>>>>>>> <
> https://github.com/apache/parquet-format/pull/624
> > > >.
> > > > > > > > > >>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>> While Divjot and others note the
> specialization for
> > > > > > > > > >> UTF-8,
> > > > > > > > > >>>>>> Parquet
> > > > > > > > > >>>>>>>>>> already
> > > > > > > > > >>>>>>>>>>> defines DOUBLE and FLOAT as IEEE encodings
> > > > > > > > > >>>>>>>>>>> <
> > > > > > > > > >>>>>>>>
> > > > > > > > > >>>>
> > > > > > > >
> > > > https://github.com/apache/parquet-format/blob/master/README.md#types
> > > > > > > > > >>>>>>>>>> .
> > > > > > > > > >>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>> Making NaN or +/-Inf illegal inside VECTOR
> would
> > > be a
> > > > > new
> > > > > > > > > >>>>>>> restriction
> > > > > > > > > >>>>>>>>> on
> > > > > > > > > >>>>>>>>>>> those same underlying physical types. Scalar
> > > > > FLOAT/DOUBLE
> > > > > > > > > >>>>> columns
> > > > > > > > > >>>>>>>> have
> > > > > > > > > >>>>>>>>>>> never had such a rule.
> > > > > > > > > >>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>> It would also not match how the systems that
> > > actually
> > > > > > > > > >>> produce
> > > > > > > > > >>>>>> these
> > > > > > > > > >>>>>>>>>> vectors
> > > > > > > > > >>>>>>>>>>> treat floats. Most of them use IEEE floats,
> e.g.
> > > > Spark,
> > > > > > > > > >>>> DuckDB,
> > > > > > > > > >>>>>>>> NumPy,
> > > > > > > > > >>>>>>>>>>> PyTorch, TensorFlow, or lower level CUDA,
> oneMKL,
> > > and
> > > > > > > > > >>>> OpenBLAS.
> > > > > > > > > >>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>> pgvector's "all elements must be finite" rule
> is an
> > > > > > > > > >> insert
> > > > > > > > > >>>>>> policy.
> > > > > > > > > >>>>>>>>> While
> > > > > > > > > >>>>>>>>>> I
> > > > > > > > > >>>>>>>>>>> think Parquet as a general-purpose format
> should
> > > > allow
> > > > > > > > > >> them
> > > > > > > > > >>>> out
> > > > > > > > > >>>>>> of
> > > > > > > > > >>>>>>>> the
> > > > > > > > > >>>>>>>>>> box.
> > > > > > > > > >>>>>>>>>>> It is a user system decision to disallow
> writing
> > > > these
> > > > > > > > > >>> values
> > > > > > > > > >>>>> or
> > > > > > > > > >>>>>> to
> > > > > > > > > >>>>>>>>> throw
> > > > > > > > > >>>>>>>>>>> errors when encountering them.
> > > > > > > > > >>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>> A type-level ban also does not keep Inf/NaN
> out of
> > > > > vector
> > > > > > > > > >>>> math.
> > > > > > > > > >>>>>>>> Finite
> > > > > > > > > >>>>>>>>>>> coordinates still overflow. For instance:
> > > magnitude =
> > > > > > > > > >>>>>> ||[FLOAT_MAX,
> > > > > > > > > >>>>>>>>>>> FLOAT_MAX]|| where every coordinate is finite
> > > > overflows
> > > > > > > > > >> to
> > > > > > > > > >>>>> +Inf.
> > > > > > > > > >>>>>>>>> PyTorch
> > > > > > > > > >>>>>>>>>>> documents the same overflow: tensor([1e20,
> > > > > 1e20]).norm()
> > > > > > > > > >> is
> > > > > > > > > >>>> inf
> > > > > > > > > >>>>>> in
> > > > > > > > > >>>>>>>>> fp32.
> > > > > > > > > >>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>> A generated MAGNITUDE(vec) column would then
> be Inf
> > > > > while
> > > > > > > > > >>> the
> > > > > > > > > >>>>>>> VECTOR
> > > > > > > > > >>>>>>>>>> itself
> > > > > > > > > >>>>>>>>>>> was required to be finite. The ban only forbids
> > > > storing
> > > > > > > > > >>>>> specials
> > > > > > > > > >>>>>> in
> > > > > > > > > >>>>>>>> the
> > > > > > > > > >>>>>>>>>>> coordinates, and does not prevent even simple
> > > > > downstream
> > > > > > > > > >>>>>>> computations
> > > > > > > > > >>>>>>>>>> from
> > > > > > > > > >>>>>>>>>>> producing them.
> > > > > > > > > >>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>> I suggest keeping VECTOR as the type and add an
> > > > > optional
> > > > > > > > > >>> bool
> > > > > > > > > >>>>>>>>>>> VectorType.finite (default false). Engines that
> > > want
> > > > > > > > > >>>>>> pgvector-like
> > > > > > > > > >>>>>>>>>>> rejection can require the flag = true. Engines
> that
> > > > use
> > > > > > > > > >>>> NumPy /
> > > > > > > > > >>>>>>>>> PyTorch /
> > > > > > > > > >>>>>>>>>>> Spark arrays can keep representable IEEE
> values.
> > > For
> > > > > the
> > > > > > > > > >>> same
> > > > > > > > > >>>>>>>> reason, I
> > > > > > > > > >>>>>>>>>> do
> > > > > > > > > >>>>>>>>>>> not favor adding yet another type called
> > > > FINITE_VECTOR.
> > > > > > > > > >>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>> Best,
> > > > > > > > > >>>>>>>>>>> Sebastian
> > > > > > > > > >>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>> On Tue, Sep 29, 2026 at 2:03 PM Divjot Arora
> via
> > > dev
> > > > <
> > > > > > > > > >>>>>>>>>>> dev@parquet.apache.org>
> > > > > > > > > >>>>>>>>>>> wrote:
> > > > > > > > > >>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>> I agree with moving forward with the proposal
> and
> > > > > > > > > >>> renaming
> > > > > > > > > >>>>>> VECTOR
> > > > > > > > > >>>>>>>> to
> > > > > > > > > >>>>>>>>>>>> FINITE_VECTOR or something
> > > > > > > > > >>>>>>>>>>>> similar if others feel that better
> communicates
> > > its
> > > > > > > > > >>>> intent. I
> > > > > > > > > >>>>>> see
> > > > > > > > > >>>>>>>>> value
> > > > > > > > > >>>>>>>>>>> in
> > > > > > > > > >>>>>>>>>>>> having a vector type that guarantees
> > > > > > > > > >>>>>>>>>>>> its values are mathematically-correct vectors
> for
> > > a
> > > > > few
> > > > > > > > > >>>>>> reasons:
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>> 1. As others have mentioned, this is similar
> to
> > > the
> > > > > > > > > >>> STRING
> > > > > > > > > >>>>>>> logical
> > > > > > > > > >>>>>>>>> type
> > > > > > > > > >>>>>>>>>>>> mandating UTF-8.
> > > > > > > > > >>>>>>>>>>>> 2. It is not possible to rely on statistics
> for
> > > > > > > > > >> detecting
> > > > > > > > > >>>> NaN
> > > > > > > > > >>>>>> or
> > > > > > > > > >>>>>>>>> +-inf
> > > > > > > > > >>>>>>>>>>>> values as all statistics are optional.
> > > > > > > > > >>>>>>>>>>>> To my knowledge, there is no precedent for
> > > requiring
> > > > > > > > > >> that
> > > > > > > > > >>>>>> writers
> > > > > > > > > >>>>>>>>> emit
> > > > > > > > > >>>>>>>>>>>> stats for specific logical types.
> > > > > > > > > >>>>>>>>>>>> 3. Without statistics, NaN/inf values could
> flow
> > > > > > > > > >> through
> > > > > > > > > >>>> the
> > > > > > > > > >>>>>>> engine
> > > > > > > > > >>>>>>>>> and
> > > > > > > > > >>>>>>>>>>>> block them for specific functionality.
> > > > > > > > > >>>>>>>>>>>> This seems to be Lance's behavior. However, I
> > > don't
> > > > > > > > > >> think
> > > > > > > > > >>>>> this
> > > > > > > > > >>>>>> is
> > > > > > > > > >>>>>>>> as
> > > > > > > > > >>>>>>>>>> easy
> > > > > > > > > >>>>>>>>>>>> as it sounds and Lance has run
> > > > > > > > > >>>>>>>>>>>> into several bugs due to this ([1] [2] for
> > > > examples).
> > > > > > > > > >>> Their
> > > > > > > > > >>>>>>> Python
> > > > > > > > > >>>>>>>>> [3]
> > > > > > > > > >>>>>>>>>>> and
> > > > > > > > > >>>>>>>>>>>> Rust [4] APIs default to erroring on
> > > > > > > > > >>>>>>>>>>>> vectors that contain NaN values.
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>> I agree that Parquet is a general-purpose
> format,
> > > > but
> > > > > I
> > > > > > > > > >>>> don't
> > > > > > > > > >>>>>>> think
> > > > > > > > > >>>>>>>>>> this
> > > > > > > > > >>>>>>>>>>>> implies the lack of domain-specific types.
> > > > > > > > > >>>>>>>>>>>> Instead, this broadens the use cases where
> Parquet
> > > > > > > > > >> would
> > > > > > > > > >>> be
> > > > > > > > > >>>>>>>>> considered
> > > > > > > > > >>>>>>>>>> a
> > > > > > > > > >>>>>>>>>>>> good fit. I also fully agree with
> > > > > > > > > >>>>>>>>>>>> continuing work on a general fixed-size list
> type
> > > if
> > > > > we
> > > > > > > > > >>>>> believe
> > > > > > > > > >>>>>>>> there
> > > > > > > > > >>>>>>>>>> are
> > > > > > > > > >>>>>>>>>>>> non-vector use cases that would benefit
> > > > > > > > > >>>>>>>>>>>> from such guarantees at the storage layer.
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>> [1]
> > > https://github.com/lance-format/lance/pull/3648
> > > > > > > > > >>>>>>>>>>>> [2]
> > > > https://github.com/lance-format/lance/issues/8607
> > > > > > > > > >>> (see
> > > > > > > > > >>>>>>> point 2
> > > > > > > > > >>>>>>>>> in
> > > > > > > > > >>>>>>>>>>>> "root
> > > > > > > > > >>>>>>>>>>>> cause")
> > > > > > > > > >>>>>>>>>>>> [3]
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>
> > > > > > > > > >>>>
> > > > > > > > > >>>
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> https://github.com/lancedb/lancedb/blob/0327618a62483feb6dca2336549845e5887cd48f/python/python/lancedb/table.py#L4807-L4911
> > > > > > > > > >>>>>>>>>>>> [4]
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>
> > > > > > > > > >>>>
> > > > > > > > > >>>
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> https://github.com/lancedb/lancedb/blob/0327618a62483feb6dca2336549845e5887cd48f/rust/lancedb/src/table/add_data.rs#L105-L110
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>> On Mon, Sep 28, 2026 at 6:19 PM Philipp
> Fischbeck
> > > > via
> > > > > > > > > >>> dev <
> > > > > > > > > >>>>>>>>>>>> dev@parquet.apache.org> wrote:
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>> I also favor moving forward with the
> proposal as
> > > > is.
> > > > > > > > > >> I
> > > > > > > > > >>>> see
> > > > > > > > > >>>>>>> strong
> > > > > > > > > >>>>>>>>>> value
> > > > > > > > > >>>>>>>>>>>> and
> > > > > > > > > >>>>>>>>>>>>> use cases for the numeric vector type, and I
> > > agree
> > > > > > > > > >> with
> > > > > > > > > >>>> the
> > > > > > > > > >>>>>>>>>>> restrictions
> > > > > > > > > >>>>>>>>>>>>> around NaN/Inf values and element types in
> > > general.
> > > > > > > > > >>>>> Precedent
> > > > > > > > > >>>>>>>>> exists
> > > > > > > > > >>>>>>>>>> in
> > > > > > > > > >>>>>>>>>>>>> other types that restrict which physical type
> > > > values
> > > > > > > > > >>> are
> > > > > > > > > >>>>>>>> logically
> > > > > > > > > >>>>>>>>>>> valid
> > > > > > > > > >>>>>>>>>>>>> (e.g., the geospatial types needing to be
> > > > > > > > > >> WKB-encoded).
> > > > > > > > > >>>> A
> > > > > > > > > >>>>>>>> general
> > > > > > > > > >>>>>>>>>>>>> fixed-size list also has value, but I think
> it
> > > > makes
> > > > > > > > > >>>> sense
> > > > > > > > > >>>>> to
> > > > > > > > > >>>>>>>>>> separate
> > > > > > > > > >>>>>>>>>>>> the
> > > > > > > > > >>>>>>>>>>>>> two types. The semantics are relevant, and I
> can
> > > > see
> > > > > > > > > >>> how
> > > > > > > > > >>>>> the
> > > > > > > > > >>>>>>> two
> > > > > > > > > >>>>>>>>>> types
> > > > > > > > > >>>>>>>>>>>>> might diverge further in the future regarding
> > > > > > > > > >>> statistics
> > > > > > > > > >>>> or
> > > > > > > > > >>>>>>>>>> encodings.
> > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>> I would be okay with renaming the vector
> type,
> > > but
> > > > it
> > > > > > > > > >>>> seems
> > > > > > > > > >>>>>>> like
> > > > > > > > > >>>>>>>>> the
> > > > > > > > > >>>>>>>>>>>> VECTOR
> > > > > > > > > >>>>>>>>>>>>> name is already used across other engines
> > > mentioned
> > > > > > > > > >> in
> > > > > > > > > >>>> this
> > > > > > > > > >>>>>>>> thread.
> > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>> Best,
> > > > > > > > > >>>>>>>>>>>>> Philipp
> > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>> On Sat, Sep 26, 2026 at 4:37 PM Gang Wu <
> > > > > > > > > >>>> ustcwg@gmail.com>
> > > > > > > > > >>>>>>>> wrote:
> > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>> To separate orthogonal concerns, I feel we
> > > should
> > > > > > > > > >>> treat
> > > > > > > > > >>>>> the
> > > > > > > > > >>>>>>>>> general
> > > > > > > > > >>>>>>>>>>>>>> FixedSizeList as the underlying physical
> layout,
> > > > > > > > > >> and
> > > > > > > > > >>>>>> separate
> > > > > > > > > >>>>>>>> it
> > > > > > > > > >>>>>>>>>> from
> > > > > > > > > >>>>>>>>>>>> the
> > > > > > > > > >>>>>>>>>>>>>> discussion of a specialized VECTOR (or
> broader
> > > ML
> > > > > > > > > >>>> tensor)
> > > > > > > > > >>>>>>>> logical
> > > > > > > > > >>>>>>>>>>> type.
> > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>> Currently, the immediate need is to define a
> > > > > > > > > >>>> specialized
> > > > > > > > > >>>>>>> VECTOR
> > > > > > > > > >>>>>>>>>> type
> > > > > > > > > >>>>>>>>>>> so
> > > > > > > > > >>>>>>>>>>>>>> that different engines and formats have a
> common
> > > > > > > > > >>>> semantic
> > > > > > > > > >>>>>>>>> contract
> > > > > > > > > >>>>>>>>>>> for
> > > > > > > > > >>>>>>>>>>>>>> interoperability.
> > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>> If the main point of contention is just the
> > > > > > > > > >>>> NaN/Inf/null
> > > > > > > > > >>>>>>>>>> limitations,
> > > > > > > > > >>>>>>>>>>>>> maybe
> > > > > > > > > >>>>>>>>>>>>>> we can resolve this by adding a
> qualification to
> > > > > > > > > >> the
> > > > > > > > > >>>> name
> > > > > > > > > >>>>>>> (like
> > > > > > > > > >>>>>>>>>>> Andrew
> > > > > > > > > >>>>>>>>>>>>>> suggested, e.g., VECTOR_EMBEDDING or
> > > > FINITE_VECTOR)
> > > > > > > > > >>> to
> > > > > > > > > >>>>>> avoid
> > > > > > > > > >>>>>>>>>>> confusion
> > > > > > > > > >>>>>>>>>>>>> with
> > > > > > > > > >>>>>>>>>>>>>> broader ML use cases. Alternatively, we
> could
> > > > > > > > > >>>> introduce a
> > > > > > > > > >>>>>>>> boolean
> > > > > > > > > >>>>>>>>>>> flag
> > > > > > > > > >>>>>>>>>>>>>> (e.g., optional bool VectorType.finite) as
> > > > proposed
> > > > > > > > > >>>>>> earlier.
> > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>> On Fri, Sep 25, 2026 at 3:52 AM Andrew Lamb
> <
> > > > > > > > > >>>>>>>>>> andrewlamb11@gmail.com>
> > > > > > > > > >>>>>>>>>>>>>> wrote:
> > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>> I am a little confused about the points of
> > > > > > > > > >>>> disagreement
> > > > > > > > > >>>>>>> here.
> > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>> Is it that the name `VECTOR` is too
> general for
> > > > > > > > > >> the
> > > > > > > > > >>>>>>> specific
> > > > > > > > > >>>>>>>>> use
> > > > > > > > > >>>>>>>>>>> case
> > > > > > > > > >>>>>>>>>>>>> of
> > > > > > > > > >>>>>>>>>>>>>>> fixed length arrays with non-finite values
> used
> > > > > > > > > >> for
> > > > > > > > > >>>>>> vector
> > > > > > > > > >>>>>>>>>>>> embeddings?
> > > > > > > > > >>>>>>>>>>>>>> If
> > > > > > > > > >>>>>>>>>>>>>>> so, maybe we could add a qualification to
> the
> > > > > > > > > >> name
> > > > > > > > > >>>>>>>>>> (FINITE_VECTOR?
> > > > > > > > > >>>>>>>>>>>>>>> FEATURE_VECTOR? VECTOR_EMBEDDING?)
> > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>> It seems like there is precedent for
> defining
> > > > > > > > > >>> logical
> > > > > > > > > >>>>>> types
> > > > > > > > > >>>>>>>>> that
> > > > > > > > > >>>>>>>>>>>>> specify
> > > > > > > > > >>>>>>>>>>>>>>> certain physical types and restrict their
> > > > > > > > > >> contents
> > > > > > > > > >>> to
> > > > > > > > > >>>>>> help
> > > > > > > > > >>>>>>>>>> readers
> > > > > > > > > >>>>>>>>>>>>>>> interpret meaning, for example string[1] or
> > > JSON;
> > > > > > > > > >>>>>> Defining
> > > > > > > > > >>>>>>>> one
> > > > > > > > > >>>>>>>>>> for
> > > > > > > > > >>>>>>>>>>>>>> vector
> > > > > > > > > >>>>>>>>>>>>>>> embeddings seems reasonable enough to me.
> > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>> Andrew
> > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>> [1]:
> > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>
> > > > > > > > > >>>>
> > > > > > > > > >>>
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#string
> > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>> On Thu, Sep 24, 2026 at 3:34 PM Daniel
> Weeks <
> > > > > > > > > >>>>>>>>> dweeks@apache.org>
> > > > > > > > > >>>>>>>>>>>>> wrote:
> > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>> Well, Parquet is not a single-purpose
> format.
> > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>> Vector stores can choose whatever policy
> fits
> > > > > > > > > >>>> their
> > > > > > > > > >>>>>>>> usage,
> > > > > > > > > >>>>>>>>>> but
> > > > > > > > > >>>>>>>>>>>>> we're
> > > > > > > > > >>>>>>>>>>>>>>>>> talking about a general-purpose file
> format
> > > > > > > > > >>> that
> > > > > > > > > >>>>> aims
> > > > > > > > > >>>>>>> to
> > > > > > > > > >>>>>>>> be
> > > > > > > > > >>>>>>>>>>>> broadly
> > > > > > > > > >>>>>>>>>>>>>>>>> applicable.
> > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>> I agree that the format is intended to be
> > > > > > > > > >> general
> > > > > > > > > >>>>>>> purpose,
> > > > > > > > > >>>>>>>>> but
> > > > > > > > > >>>>>>>>>>>> types
> > > > > > > > > >>>>>>>>>>>>>>> within
> > > > > > > > > >>>>>>>>>>>>>>>> the format define specific representation
> and
> > > > > > > > > >>>>> semantics
> > > > > > > > > >>>>>>> to
> > > > > > > > > >>>>>>>>>>> achieve
> > > > > > > > > >>>>>>>>>>>>> that
> > > > > > > > > >>>>>>>>>>>>>>>> goal.
> > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>> "Vector" has a more general sense than
> ML and
> > > > > > > > > >>>> it's
> > > > > > > > > >>>>>>> quite
> > > > > > > > > >>>>>>>>>>> trivial
> > > > > > > > > >>>>>>>>>>>> to
> > > > > > > > > >>>>>>>>>>>>>>>>> produce vectors with non-finite values
> using
> > > > > > > > > >>> e.g.
> > > > > > > > > >>>>>> NumPy
> > > > > > > > > >>>>>>>> or
> > > > > > > > > >>>>>>>>>>> Pandas
> > > > > > > > > >>>>>>>>>>>>> (or
> > > > > > > > > >>>>>>>>>>>>>>>>> any other piece of dataframe or
> scientific
> > > > > > > > > >>>>> computing
> > > > > > > > > >>>>>>>>>> software).
> > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>> The justification for restricting
> non-finite
> > > > > > > > > >>>> values
> > > > > > > > > >>>>> is
> > > > > > > > > >>>>>>>>>> directly
> > > > > > > > > >>>>>>>>>>>>>> related
> > > > > > > > > >>>>>>>>>>>>>>> to
> > > > > > > > > >>>>>>>>>>>>>>>> the semantic meaning of a vector.  Vectors
> > > have
> > > > > > > > > >>>>>> specific
> > > > > > > > > >>>>>>>>>>>>>> characteristics
> > > > > > > > > >>>>>>>>>>>>>>>> (dimension, magnitude, direction, etc).
> > > > > > > > > >>> Non-finite
> > > > > > > > > >>>>>>> values
> > > > > > > > > >>>>>>>>> are
> > > > > > > > > >>>>>>>>>>>>> invalid
> > > > > > > > > >>>>>>>>>>>>>>> and
> > > > > > > > > >>>>>>>>>>>>>>>> break those characteristics.  I'd liken
> it to
> > > > > > > > > >>>> storing
> > > > > > > > > >>>>>>>>> non-utf8
> > > > > > > > > >>>>>>>>>>>> values
> > > > > > > > > >>>>>>>>>>>>>> in
> > > > > > > > > >>>>>>>>>>>>>>> a
> > > > > > > > > >>>>>>>>>>>>>>>> string and still calling it a string.
> > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>> That also differentiates a vector from a
> fixed
> > > > > > > > > >>>> length
> > > > > > > > > >>>>>>> list,
> > > > > > > > > >>>>>>>>>> which
> > > > > > > > > >>>>>>>>>>>> is
> > > > > > > > > >>>>>>>>>>>>>> more
> > > > > > > > > >>>>>>>>>>>>>>>> general and does not carry the same
> semantic
> > > > > > > > > >>>> meaning.
> > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>> I also don't think we need to bundle all
> > > > > > > > > >>> similarly
> > > > > > > > > >>>>>> shaped
> > > > > > > > > >>>>>>>> use
> > > > > > > > > >>>>>>>>>>> cases
> > > > > > > > > >>>>>>>>>>>>>> into
> > > > > > > > > >>>>>>>>>>>>>>>> the same solution (e.g. flxed size list).
> > > > > > > > > >> Vector
> > > > > > > > > >>>> is
> > > > > > > > > >>>>> a
> > > > > > > > > >>>>>>> well
> > > > > > > > > >>>>>>>>>>>>> established
> > > > > > > > > >>>>>>>>>>>>>>>> data type at this point and aligns with
> the
> > > > > > > > > >>> general
> > > > > > > > > >>>>>>> purpose
> > > > > > > > > >>>>>>>>>>> nature
> > > > > > > > > >>>>>>>>>>>> of
> > > > > > > > > >>>>>>>>>>>>>> the
> > > > > > > > > >>>>>>>>>>>>>>>> format (many other formats and databases
> > > > > > > > > >> natively
> > > > > > > > > >>>>>> support
> > > > > > > > > >>>>>>>>>> this).
> > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>> -Dan
> > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>> On Thu, Sep 24, 2026 at 11:55 AM Antoine
> > > > > > > > > >> Pitrou <
> > > > > > > > > >>>>>>>>>>>> antoine@python.org>
> > > > > > > > > >>>>>>>>>>>>>>>> wrote:
> > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>> Le 24/09/2026 à 20:11, Russell Spitzer a
> > > > > > > > > >> écrit
> > > > > > > > > >>> :
> > > > > > > > > >>>>>>>>>>>>>>>>>> I think something I haven't seen yet is
> the
> > > > > > > > > >>>> clear
> > > > > > > > > >>>>>>>>> use-case
> > > > > > > > > >>>>>>>>>>> for
> > > > > > > > > >>>>>>>>>>>>>>>> non-finite
> > > > > > > > > >>>>>>>>>>>>>>>>>> values within a vector type. There has
> > > > > > > > > >> been a
> > > > > > > > > >>>> lot
> > > > > > > > > >>>>>> of
> > > > > > > > > >>>>>>>>>>>> theoretical
> > > > > > > > > >>>>>>>>>>>>>>>>> discussion
> > > > > > > > > >>>>>>>>>>>>>>>>>> here about use cases but what toolings
> > > > > > > > > >>>> currently
> > > > > > > > > >>>>>>>> produce
> > > > > > > > > >>>>>>>>>>> these
> > > > > > > > > >>>>>>>>>>>>>>> vectors
> > > > > > > > > >>>>>>>>>>>>>>>>> and
> > > > > > > > > >>>>>>>>>>>>>>>>>> how do they function in downstream
> engines?
> > > > > > > > > >>> I'd
> > > > > > > > > >>>>>> like
> > > > > > > > > >>>>>>> to
> > > > > > > > > >>>>>>>>>> limit
> > > > > > > > > >>>>>>>>>>>>>>>> introducing
> > > > > > > > > >>>>>>>>>>>>>>>>>> types that have a wide range of values
> > > > > > > > > >> which
> > > > > > > > > >>>> are
> > > > > > > > > >>>>>>> simply
> > > > > > > > > >>>>>>>>> are
> > > > > > > > > >>>>>>>>>>>>> failure
> > > > > > > > > >>>>>>>>>>>>>>>> modes
> > > > > > > > > >>>>>>>>>>>>>>>>>> of different flavors
> > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>> "Vector" has a more general sense than
> ML and
> > > > > > > > > >>>> it's
> > > > > > > > > >>>>>>> quite
> > > > > > > > > >>>>>>>>>>> trivial
> > > > > > > > > >>>>>>>>>>>> to
> > > > > > > > > >>>>>>>>>>>>>>>>> produce vectors with non-finite values
> using
> > > > > > > > > >>> e.g.
> > > > > > > > > >>>>>> NumPy
> > > > > > > > > >>>>>>>> or
> > > > > > > > > >>>>>>>>>>> Pandas
> > > > > > > > > >>>>>>>>>>>>> (or
> > > > > > > > > >>>>>>>>>>>>>>>>> any other piece of dataframe or
> scientific
> > > > > > > > > >>>>> computing
> > > > > > > > > >>>>>>>>>> software).
> > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>> It's not like the general concept of a
> vector
> > > > > > > > > >>> is
> > > > > > > > > >>>> a
> > > > > > > > > >>>>>> IETF
> > > > > > > > > >>>>>>>>>>> standard
> > > > > > > > > >>>>>>>>>>>> or
> > > > > > > > > >>>>>>>>>>>>>>>>> anything, so analogies with JSON or UTF8
> > > > > > > > > >> (where
> > > > > > > > > >>>> the
> > > > > > > > > >>>>>>>> values
> > > > > > > > > >>>>>>>>>> are
> > > > > > > > > >>>>>>>>>>>>>> expected
> > > > > > > > > >>>>>>>>>>>>>>>>> to be valid according to the respective
> > > > > > > > > >> specs)
> > > > > > > > > >>> do
> > > > > > > > > >>>>> not
> > > > > > > > > >>>>>>>>> apply.
> > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>> This discussion would IMHO have a better
> > > > > > > > > >> chance
> > > > > > > > > >>>> of
> > > > > > > > > >>>>>>>>> converging
> > > > > > > > > >>>>>>>>>>> if
> > > > > > > > > >>>>>>>>>>>> we
> > > > > > > > > >>>>>>>>>>>>>>>>> separated the two orthogonal concerns of:
> > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>> 1. representing fixed-length list/vector
> data
> > > > > > > > > >>> in
> > > > > > > > > >>>>>>> Parquet
> > > > > > > > > >>>>>>>>>> (with
> > > > > > > > > >>>>>>>>>>> a
> > > > > > > > > >>>>>>>>>>>>>>>>> potential limitation to non-nested types)
> > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>> 2. representing the information that a
> given
> > > > > > > > > >>>> column
> > > > > > > > > >>>>>> (or
> > > > > > > > > >>>>>>>>>> column
> > > > > > > > > >>>>>>>>>>>>> chunk,
> > > > > > > > > >>>>>>>>>>>>>>>>> etc.) does not contain non-finite values
> > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>> ... both of which are useful on their own
> > > > > > > > > >>> right,
> > > > > > > > > >>>>> but
> > > > > > > > > >>>>>> do
> > > > > > > > > >>>>>>>> not
> > > > > > > > > >>>>>>>>>>> have
> > > > > > > > > >>>>>>>>>>>> to
> > > > > > > > > >>>>>>>>>>>>>> be
> > > > > > > > > >>>>>>>>>>>>>>>>> conflated and bundled together.
> > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>> Regards
> > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>> Antoine.
> > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>
> > > > > > > > > >>>>
> > > > > > > > > >>>
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
>

Re: [DISCUSS] Numeric Vector type: physical representation (outcome of focused call)

[Curt Hagenlocher <cu...@hagenlocher.org>]

> I'm not aware of databases with finite floating point types.

Microsoft SQL Server, MySQL and SAP ASE (née Sybase) do not allow
storing non-finite floating-point values in their binary
floating-point columns. They don't have a separate type which does
allow this.

On Wed, Oct 21, 2026 at 3:30 PM Ryan Blue <bl...@apache.org> wrote:
>
> Jan,
>
> I agree that we _could_ model vectors as an orthogonal finiteness property
> for elements in a fixed-length list. I don't know how useful that would be.
> I've not had anyone ask for that, and I'm not aware of databases with
> finite floating point types. I think it makes more sense to move forward
> with the existing FINITE_VECTOR that is proposed.
>
> For the logical type argument, it sounds like the issue is that you want to
> fall back to readers treating vectors as LIST. I don't see a good way to
> make that happen, but it also isn't needed by the table format use case
> because we are introducing this as a new type. Do you have a use case where
> you are going to mix FINITE_VECTOR with LIST files and this will cause a
> problem?
>
> I don't think that we've had an expectation for logical types in the past
> that they produce a particular in-memory representation. Maybe we could
> allow setting both `LIST` and `FINITE_VECTOR` logical types at the same
> time? This kind of compatibility hasn't been a priority as far as I'm
> aware, but if it is important to you we can consider it.
>
> Ryan
>
> On Wed, Oct 21, 2026 at 1:13 PM Jan Finis <jp...@gmail.com> wrote:
>
> >  assuming what will be proposed is a new logical type FINITE_VECTOR that
> > annotates the same column that would have been annotated as LIST before:
> >
> > What I find somewhat suboptimal of FINITE_VECTOR is that finiteness is part
> > of the vector type. Isn't the finiteness rather a property of the contained
> > floats? Such a modeling would be more composable, as it would for example
> > also allow to have normal (non-vector) float columns that are restricted to
> > be finite. So, if we do think that we want finiteness in the type system, I
> > would rather cut it into two parts: A fixed size vector concept, and a new
> > way of labeling float columns finite. The vector type would then just boil
> > down to be a fixed size list / we could just give the list type an optional
> > fixed size field. Then, the column contained within it would be a float
> > tagged as finite.
> >
> > It was mentioned that a logical type annotating a list of float would be
> > forward compatible, but - depending on the modeling - that is not
> > necessarily true: as logical types in Parquet do not support subtyping, a
> > column cannot be a list of float and a (finite) vector at the same time,
> > but only one of the two. It depends on how the type will be modeled in
> > thrift in the end, but if it is a new union member of `LogicalType`, then
> > legacy engines will not understand that such a vector can also just be
> > interpreted as a list of floats. They will just see a nested column type of
> > a logical type they don't understand, so they will likely reject reading
> > this column (at least that's what the engine I'm maintaining would be
> > doing).
> >
> > In contrast, if we just give the list type an optional fixed size and float
> > an optional finiteness, then the resulting thrift is actually fully forward
> > compatible: Legacy engines will just ignore the fixed length and treat the
> > list as variable length and they will ignore the finiteness and treat the
> > element type as float.
> >
> > (A side note, which most people here may already be aware of, but it's good
> > to repeat it: The not-NULL property doesn't need new type support at all.
> > The normal type system of list already allows setting the contained element
> > type to be optional or required. If it is set to be required, then NULLs
> > are not allowed.)
> >
> > Cheers,
> > Jan
> >
> >
> >
> >
> >
> > Am Di., 20. Okt. 2026 um 20:37 Uhr schrieb Russell Spitzer <
> > russell.spitzer@gmail.com>:
> >
> > > I support proceeding with the two types. I honestly think the cost of
> > > adding new logical types is overestimated. If we have as many interested
> > > parties as have appeared in this thread we already know there is a large
> > > group of engineers ready to do the work and engines ready to consume (and
> > > produce) the type. The fact that implementations currently use
> > heuristics,
> > > sampling (I loved Gunnar's talk on his hardwood work on this at CoC
> > > Glasgow) or custom annotations to achieve the same performance benefits
> > > makes me think we should just create the type and move on.
> > >
> > > On Mon, Oct 19, 2026 at 7:21 PM Ryan Blue <bl...@apache.org> wrote:
> > >
> > > > An annotation on list<float> is useful because:
> > > > - We want to correctly annotate what is being stored
> > > > - We want to unlock optimizations when each non-null list has the same
> > > > number of elements
> > > > - We want to have writer guarantees
> > > >
> > > > I don't think that logical type compatibility is a good argument. We
> > want
> > > > to avoid a case where we have to treat all binary like a string. And
> > > > because we are introducing this in table formats after standardizing it
> > > > here, we can do that reliably.
> > > >
> > > > As I said in my last email, I think there's a difference between
> > > finiteness
> > > > and general "invariants" you might want. Vectors aren't vectors if they
> > > > don't have a length and a direction. Other invariants don't work the
> > same
> > > > way. A zero-length vector is still a valid vector and can be used, just
> > > not
> > > > for certain distance metrics.
> > > >
> > > > I also want to clarify that when we looked into this more closely and
> > > > recognized the mismatch between the two separate use cases, I offered
> > to
> > > > split out the finite vector work so that we didn't cause any headaches
> > > for
> > > > people who are interested in fixed-length list. It sounds like not
> > > > splitting out the use cases is causing a lot of headache. Vectors for
> > our
> > > > use case need a finiteness guarantee at write time, and fixed-length
> > > lists
> > > > need to store NaN and other values.
> > > >
> > > > It's okay to have two types and we should for two different use cases.
> > > > Otherwise, we are forcing one to make an unnecessary compromise. I
> > don't
> > > > want to force everyone to exclude NaN and infinite values from
> > > fixed-length
> > > > lists. Seems reasonable that we should also be able to have vectors
> > that
> > > > exclude those values, since that is how they are used and we have to
> > also
> > > > guarantee it for exchanging data with other existing databases.
> > > >
> > > > Ryan
> > > >
> > > > On Mon, Oct 19, 2026 at 3:12 AM Will Edwards via dev <
> > > > dev@parquet.apache.org>
> > > > wrote:
> > > >
> > > > > It reads well. Three small fixes:
> > > > >
> > > > > 1. **"back in May or so"** is ambiguous. The
> > > fixed-size-list-for-tensors
> > > > > framing is from Rok's opening post in May 2024. The May 2026 design
> > doc
> > > > was
> > > > > already mostly about performance and layout. "Originally" avoids
> > anyone
> > > > > correcting the date.
> > > > > 2. **"Spark's sparse feature vectors routinely use NaN"** isn't quite
> > > > > right. The NaN-for-missing behaviour comes from `VectorAssembler` and
> > > > > `Imputer` and applies to Spark ML vectors generally, dense or sparse.
> > > > > "Spark's feature vectors, dense and sparse," fixes it.
> > > > > 3. There are typos: "Iirc" should be "IIRC", and "Spark, BigQuery and
> > > > > DuckDB and pretty much…" has a doubled "and".
> > > > >
> > > > > The last sentence of the invariants paragraph could be clearer.
> > > "Although
> > > > > that doesn't mean readers shouldn't check on read" has a double
> > > negative
> > > > > and is easy to misread. "Though careful readers will still check on
> > > read"
> > > > > is simpler.
> > > > >
> > > > > With those applied:
> > > > >
> > > > > ---
> > > > >
> > > > > Hi all,
> > > > >
> > > > > The scope of this discussion keeps moving :). IIRC it originally was
> > > > about
> > > > > fixed-size lists for ML tensors, became a performance question, and
> > is
> > > > now
> > > > > about whether vectors are mathematically valid. Each step made sense,
> > > > but I
> > > > > don't think we have consensus on where we've ended up?
> > > > >
> > > > > I think there is much useful work in what we left behind. Many ML
> > > vectors
> > > > > aren't embeddings for search. E.g. Spark's feature vectors, dense and
> > > > > sparse, routinely use NaN to mark missing values, and quantised
> > formats
> > > > > pack codes with scales in ways Parquet can't describe today.
> > > ML-oriented
> > > > > formats like Lance are moving towards tensors, quantisation and
> > > indexes.
> > > > A
> > > > > type that told engines how to interpret that kind of data would
> > enable
> > > > new
> > > > > things. An annotation on LIST<FLOAT> doesn't.
> > > > >
> > > > > My big concern with the current focus on embeddings needing a logical
> > > > type
> > > > > is compatibility. Today LIST<FLOAT> is the common format for
> > > embeddings,
> > > > > and Spark, BigQuery, DuckDB and pretty much every other library I can
> > > > think
> > > > > of offer distance functions on plain lists, so any engine can use any
> > > > file.
> > > > > If engines start requiring a VECTOR annotation, existing files and
> > > > anything
> > > > > rewritten by a tool that doesn't know the type stop working. If they
> > > > don't
> > > > > require it, the annotation adds nothing. We've seen this before with
> > > > > strings written without the UTF8 annotation, which engines still have
> > > to
> > > > > cope with years later.
> > > > >
> > > > > On invariants, I agree some users may want the writer to check what
> > > they
> > > > > write as they write it, but that can be a writer option rather than a
> > > > type.
> > > > > Readers that want evidence could get it from extra statistics or
> > > > metadata,
> > > > > though careful readers will still check on read.
> > > > >
> > > > > If a finite fixed-length vector contract is useful to table formats,
> > > > > perhaps Iceberg is the place for it, with Parquet storing plain
> > lists.
> > > > I'd
> > > > > raise the same worry there, though: if tables start rejecting
> > untagged
> > > > > columns, we've just moved the split up a layer.
> > > > >
> > > > > Best,
> > > > > Will
> > > > >
> > > > > On Sat, 17 Oct 2026 at 00:10, Ryan Blue <bl...@apache.org> wrote:
> > > > >
> > > > > > I agree with much of what Dan said, but I'll add some to this
> > thread
> > > as
> > > > > > well.
> > > > > >
> > > > > > First, I want to address the arguments that were pointed out:
> > > > > > 1. Other vector databases have finiteness requirements
> > > > > > 2. "Vectors are the hot stuff right now" -- let's jump on the
> > > bandwagon
> > > > > > 3. A general solution will take too long, finiteness will make
> > faster
> > > > > > progress
> > > > > > 4. To give a guarantee to readers that no INF/NaN/NULL will be in
> > the
> > > > > > vector
> > > > > >
> > > > > > I don't think these represent our actual motivation or position,
> > > which
> > > > > > makes them function as a straw-man. This is probably not what was
> > > > > > intended, but I want to make sure that the misunderstanding
> > > > > > doesn't continue. For one thing, we aren't saying that we need
> > finite
> > > > > > vectors because they are "hot stuff".
> > > > > >
> > > > > > We want to build tools that use them on top of Parquet. I think it
> > > > should
> > > > > > also be clear that we aren't advocating for finite elements blindly
> > > > just
> > > > > > because other engines do (cargo cult design!). There are real
> > > > > > interoperability issues if we choose to go a different direction so
> > > > this
> > > > > is
> > > > > > a "Chesterton's Fence" situation. It's important to understand why
> > > > > > those databases
> > > > > > made that choice. I've also pointed this out as confirmation that
> > > we're
> > > > > > going down a reasonable and established path.
> > > > > >
> > > > > > The third argument sounds like a simple misunderstanding. We're
> > > > pursuing
> > > > > a
> > > > > > logical type right now because that's forward compatible and we
> > want
> > > to
> > > > > > label the data properly and set expectations for the type. Physical
> > > > type
> > > > > > changes break forward compatibility and need to be released in a
> > new
> > > > > format
> > > > > > version . I think this is the source of the confusion since it has
> > > been
> > > > > in
> > > > > > the same discussion, but the choice about finiteness is separate.
> > > > > >
> > > > > > That brings us to the last argument, whether there is value in
> > > writers
> > > > > > participating in guarantees. I think there's value in adding
> > > > requirements
> > > > > > for writers. It's up to readers how they want to use those
> > > > requirements,
> > > > > > but a writer requirement allows a policy of "garbage in garbage
> > out"
> > > > > rather
> > > > > > than being forced to handle bad data. Why would a reader need to
> > run
> > > > > > validation on every element just in case a writer had a bug? And
> > why
> > > > > should
> > > > > > the potential for writer bugs mean that it has to be the reader's
> > > > > problem?
> > > > > > If a writer produces bad data, we can absolutely reject it and tell
> > > the
> > > > > > writer to fix the bug.
> > > > > >
> > > > > > I'm also puzzled by the assertion that writer guarantees have
> > value,
> > > > but
> > > > > > only if there are other, more specific guarantees in addition. This
> > > > seems
> > > > > > to cede the point that a contract between readers and writers makes
> > > > > sense,
> > > > > > but only if it is a _specific_ contract. But the proposed additions
> > > > don't
> > > > > > make sense in the same way.
> > > > > >
> > > > > > Take, for example, the suggestion to also disallow zero-length
> > > vectors.
> > > > > At
> > > > > > a glance this seems similar, but I think it assumes a specific
> > > context
> > > > > and
> > > > > > is better suited to be a constraint for some use cases rather than
> > > part
> > > > > of
> > > > > > what is considered valid for the type. That's because zero-length
> > > > vectors
> > > > > > are valid vectors with a length. You can calculate Euclidean or
> > > > Manhattan
> > > > > > distance and use them in many contexts. They don't work if you're
> > > > > > normalizing to a length of 1, but it's a choice to normalize and
> > use
> > > a
> > > > > dot
> > > > > > product to calculate cosine distance. And at this point, we're
> > > talking
> > > > > > about the usefulness of restrictions for the type, not a Platonic
> > > > ideal.
> > > > > >
> > > > > > I think the restriction that makes the most sense is that vectors
> > > have
> > > > a
> > > > > > length and can be compared in distance calculations. 0-length
> > vectors
> > > > are
> > > > > > valid because they point somewhere and have a length. We shouldn't
> > > > > restrict
> > > > > > uses to only cosine distance over normalized vectors. Those are use
> > > > case
> > > > > > dependent choices and there are other valid ones. I also don't
> > think
> > > > that
> > > > > > the model used to produce a vector is part of the vector type, just
> > > > like
> > > > > > the method used to produce a metric doesn't define its type: cosine
> > > > > > similarity produces values between -1 and 1, but we store it in a
> > > > > `float`,
> > > > > > not a `float(cosine)`.
> > > > > >
> > > > > > Why does it make sense to have a finiteness restriction as part of
> > > the
> > > > > > type? Because that's what a (mathematical) vector is. A simple
> > search
> > > > > gives
> > > > > > a pretty common definition: "A vector is an object that has both
> > > > > magnitude
> > > > > > and a direction". If a dimension/element is NaN, null, or infinite,
> > > the
> > > > > > vector doesn't point anywhere and it doesn't have a magnitude. Not
> > > only
> > > > > can
> > > > > > it not be used, it doesn't meet the most basic definition of the
> > > type.
> > > > > >
> > > > > > Ryan
> > > > > >
> > > > > > On Thu, Oct 15, 2026 at 1:07 AM Jan Finis <jp...@gmail.com>
> > wrote:
> > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > Daniel,
> > > > > > >
> > > > > > > alright, so no physical optimizations intended.  We only want
> > this
> > > > for
> > > > > > > semantic reasons, not for any significant physical optimization.
> > If
> > > > > we're
> > > > > > > all on the same page here, that is good 👍.
> > > > > > >
> > > > > > > To phrase my remaining problem with this more concretely:
> > > > > > > *Let's say we add this type. What cool new stuff can we now do
> > that
> > > > we
> > > > > > > couldn't do before?*
> > > > > > > That is the litmus test that a type has to pass to be useful
> > IMHO.
> > > > And
> > > > > > here
> > > > > > > I can't come up with a tangible use case that wasn't possible
> > > before.
> > > > > > > Especially if we one day have fixed size lists in Parquet, the
> > > > > remaining
> > > > > > > niche for this type becomes so small, that the added complexity
> > > isn't
> > > > > > worth
> > > > > > > the benefit IMHO.
> > > > > > >
> > > > > > > As I guess most people on here are active maintainers of a
> > Parquet
> > > > > > > implementation: Concretely, what would you be using this type for
> > > if
> > > > we
> > > > > > had
> > > > > > > it? Would you allow math only on fields tagged with this type?
> > What
> > > > if
> > > > > > that
> > > > > > > math yields INF or NaN? Not allowed?
> > > > > > >
> > > > > > > Especially if we also discuss having both VECTOR and
> > FINITE_VECTOR.
> > > > How
> > > > > > > will you guys wire this up with the type systems of your engines?
> > > Do
> > > > > your
> > > > > > > engines also have two types here? Do you type-decay a
> > FINITE_VECTOR
> > > > to
> > > > > a
> > > > > > > VECTOR once someone does a multiplication that could yield INF?
> > By
> > > > > > itself,
> > > > > > > a FINITE_VECTOR sounds good. Holistically looked at from a type
> > > > system
> > > > > > > perspective, it comes with a lot of complexity. Do the benefits
> > > > really
> > > > > > > outweigh the complexity here? That's why I would rather have the
> > > bare
> > > > > > > minimum: A fixed size array (list) of floats or at least just a
> > > > vector
> > > > > > > without finite. Then all these detailed questions go away.
> > > > > > >
> > > > > > > I'll take this one step further... Why is there any debate about
> > > this
> > > > > at
> > > > > > > > all? Why can't a type simply be added to the *specification* as
> > > an
> > > > > > > > administrative detail and be done?
> > > > > > >
> > > > > > >
> > > > > > > Andrew,
> > > > > > >
> > > > > > > Parquet is supposed to be a lingua franca of data. The
> > > specification
> > > > is
> > > > > > the
> > > > > > > authoritative truth for this language and all implementations are
> > > > > > expected
> > > > > > > to implement as much as possible from this spec, so anything in
> > the
> > > > > spec
> > > > > > > does causes implementation effort. So just adding anything to
> > this
> > > > spec
> > > > > > > without debate is not a good idea. It is even the other way
> > around:
> > > > We
> > > > > > need
> > > > > > > to have way higher rigor with the spec than we need to have with
> > > any
> > > > > > > implementations. Implementations can be changed, the spec needs
> > to
> > > be
> > > > > > > stable.
> > > > > > >
> > > > > > > A tag-only type that doesn't come with any restrictions can be
> > > added
> > > > > with
> > > > > > > little effort. But here we want to tie restrictions to it (no
> > > > NaN/Inf),
> > > > > > so
> > > > > > > we would expect implementations to enforce these. So there is
> > some
> > > > > > > implementation effort expected here.
> > > > > > >
> > > > > > > Also, even if we only wanted a tag type, I would still argue that
> > > you
> > > > > > > shouldn't add those without a debate. What if A wants XML, B
> > wants
> > > > > YAML,
> > > > > > C
> > > > > > > wants quaternions, D wants address, E wants phone number. If we
> > > just
> > > > > > added
> > > > > > > any types that are "just tags", Parquet logical types would just
> > > > > become a
> > > > > > > trash dump of any type that anyone ever thought to be useful. So
> > > > > rigorous
> > > > > > > assessment of each new type candidate is necessary.
> > > > > > >
> > > > > > > Finally, many engines couple parquet with another type system
> > > (e.g.,
> > > > > > Arrow
> > > > > > > or SQL). So even with a tag only type, you at least still have
> > the
> > > > > > > implementation burden of wiring this up with your engine's type
> > > > system.
> > > > > > >
> > > > > > > So, just because logical types are "just" logical, that doesn't
> > > mean
> > > > we
> > > > > > can
> > > > > > > add them without any rigor.
> > > > > > >
> > > > > > >
> > > > > > > A few naive questions:
> > > > > > > > 1. Would a physical definition with NANs & Inf+/- allow
> > efficient
> > > > > > > > encodings like ALP?
> > > > > > > > 2. If the physical representation allows values that make
> > > encodings
> > > > > > like
> > > > > > > > ALP impossible, can a tighter Logical type allow different
> > > > encodings?
> > > > > > > > 3. Can we RLE or binpack a vector?
> > > > > > >
> > > > > > >
> > > > > > > The simple general answer here is that all these things do not
> > > > require
> > > > > a
> > > > > > > logical type. Writers can already do any such optimization if the
> > > > > actual
> > > > > > > data contains no NaNs & Infs. All these physical optimizations
> > just
> > > > > > require
> > > > > > > no actual NaNs/Infs to be present in a page, they don't need a
> > > > > guarantee
> > > > > > > that the whole file won't contain any.
> > > > > > >
> > > > > > > Cheers,
> > > > > > > Jan
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Am Do., 15. Okt. 2026 um 04:28 Uhr schrieb <
> > > kurtiscwright@gmail.com
> > > > >:
> > > > > > >
> > > > > > > > Hi Everyone,
> > > > > > > >
> > > > > > > > Glad to see a Vector type being added. I have a few use cases I
> > > > want
> > > > > to
> > > > > > > > pursue once we have an agreed upon definition.
> > > > > > > >
> > > > > > > > In regards to that definition discussion. I feel comfortable
> > with
> > > > the
> > > > > > > idea
> > > > > > > > of the semantic meaning Dan raises though I can’t shake a
> > slight
> > > > > > > discomfort
> > > > > > > > (maybe FOMO) with limiting values in the physical definition.
> > > > > > > >
> > > > > > > > A few naive questions:
> > > > > > > > 1. Would a physical definition with NANs & Inf+/- allow
> > efficient
> > > > > > > > encodings like ALP?
> > > > > > > > 2. If the physical representation allows values that make
> > > encodings
> > > > > > like
> > > > > > > > ALP impossible, can a tighter Logical type allow different
> > > > encodings?
> > > > > > > > 3. Can we RLE or binpack a vector?
> > > > > > > >
> > > > > > > > Best Regards,
> > > > > > > > Kurtis C. Wright
> > > > > > > > > On Oct 14, 2026, at 12:45, Daniel Weeks <dw...@apache.org>
> > > > wrote:
> > > > > > > > >
> > > > > > > > > Hey Jan,
> > > > > > > > >
> > > > > > > > > There's a lot that I disagree with respect to the framing of
> > > > these
> > > > > > > > > arguments and how we should be thinking about introducing
> > > vector
> > > > > as a
> > > > > > > > > logical type.  There's a repeated tendency to let the
> > physical
> > > > > > > > > representation or performance drive discussion around types
> > as
> > > > > > opposed
> > > > > > > to
> > > > > > > > > what it represents.
> > > > > > > > >
> > > > > > > > > In parquet, establishing semantic meaning is the purpose of
> > > > logical
> > > > > > > > types.
> > > > > > > > > The logical type separates how to interpret a value from its
> > > > > physical
> > > > > > > > > representation.  We can represent a timestamp with INT64 or
> > > even
> > > > a
> > > > > > > string
> > > > > > > > > value, so why have a type for it? Even though a
> > > FIXED_SIZED_LIST
> > > > > and
> > > > > > > > VECTOR
> > > > > > > > > may have the exact same representation and performance, they
> > > have
> > > > > > > > different
> > > > > > > > > semantic meaning.  Databases and engines across the ecosystem
> > > > > already
> > > > > > > > > recognize this differentiation and have separate
> > > representations.
> > > > > > The
> > > > > > > > > choice to separate these isn't because the discussion is too
> > > > > tedious;
> > > > > > > > > they're fundamentally different things.
> > > > > > > > >
> > > > > > > > > The finite requirement is what makes a vector a reflection of
> > > > what
> > > > > it
> > > > > > > > > represents.  As we've discussed in the thread, non-finite
> > > values
> > > > in
> > > > > > the
> > > > > > > > > representation violate the properties of a vector and the
> > value
> > > > > > ceases
> > > > > > > to
> > > > > > > > > be a vector.  On the other hand, a 0 vector is still a vector
> > > > (you
> > > > > > can
> > > > > > > > > argue the utility of it, but it is a valid vector).
> > > > > Implementations
> > > > > > > can
> > > > > > > > > choose what they validate or how to handle execution errors;
> > > > that's
> > > > > > an
> > > > > > > > > implementation concern, not a requirement.  Operations
> > > performed
> > > > > on a
> > > > > > > > > vector depend on the context in which they are used, but the
> > > > > > > > representation
> > > > > > > > > doesn't need to solve scenarios where that application
> > doesn't
> > > > make
> > > > > > > > sense.
> > > > > > > > >
> > > > > > > > > The guarantees or context needed to use a vector do not need
> > to
> > > > be
> > > > > > > > inherent
> > > > > > > > > in the type for it to be a useful type.  We don't need the
> > > > currency
> > > > > > > used
> > > > > > > > > for decimal or whether a floating point is a normalized value
> > > as
> > > > > part
> > > > > > > of
> > > > > > > > > the type to make use of those types.  This conflates the use
> > > > cases
> > > > > > with
> > > > > > > > the
> > > > > > > > > type system and unnecessarily attempts to draw the context
> > into
> > > > the
> > > > > > > type
> > > > > > > > > definition.  Similarly, you don't need to capture
> > > normalization,
> > > > > > model,
> > > > > > > > > etc. as part of the type for it to be useful.
> > > > > > > > >
> > > > > > > > > So the reason we want to add a vector type (or FINITE_VECTOR)
> > > is
> > > > > > > because
> > > > > > > > we
> > > > > > > > > want to store vectors with the semantic meaning and
> > > requirements
> > > > > > based
> > > > > > > on
> > > > > > > > > what they represent.
> > > > > > > > >
> > > > > > > > > -Dan
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >> On Wed, Oct 14, 2026 at 1:22 AM Jan Finis <
> > jpfinis@gmail.com>
> > > > > > wrote:
> > > > > > > > >>
> > > > > > > > >> A bit late to the party, but here we go:
> > > > > > > > >>
> > > > > > > > >> I have the strong sentiment that the FINITE_VECTOR type as
> > it
> > > is
> > > > > > > > currently
> > > > > > > > >> being discussed will not be very useful and just increase
> > > > > > > implementation
> > > > > > > > >> complexity. The arguments for the type are all quite weak
> > > > (details
> > > > > > see
> > > > > > > > >> below). The FINITE part doesn't seem to yield any value and
> > is
> > > > > not a
> > > > > > > > useful
> > > > > > > > >> restriction by itself. The vector (i.e. fixed length) part
> > > > should
> > > > > > > > rather be
> > > > > > > > >> generalized instead of making a one-off thing. In general,
> > the
> > > > > > > existing
> > > > > > > > >> types are already working well for uncompressed vectors. The
> > > > real
> > > > > > > > challenge
> > > > > > > > >> that would bring real benefits is supporting quantization.
> > > This
> > > > is
> > > > > > > where
> > > > > > > > >> you get orders of magnitude improvements to storage space
> > and
> > > > > > > processing
> > > > > > > > >> speed.
> > > > > > > > >>
> > > > > > > > >> Let's first discuss the FINITE part. So far, the only
> > > arguments
> > > > I
> > > > > > have
> > > > > > > > read
> > > > > > > > >> for this are:
> > > > > > > > >>
> > > > > > > > >> 1. many other vector databases have it
> > > > > > > > >> 2. vectors are the hot stuff right now, so Parquet better
> > > boast
> > > > > > having
> > > > > > > > >> "native" type support for vectors
> > > > > > > > >> 3. by focussing on this common special case, we can make
> > rapid
> > > > > > > progress,
> > > > > > > > >> while a general solution would take too long to specify
> > > > > > > > >> 4. to give a guarantee to readers that no INF/NaN/NULL will
> > be
> > > > in
> > > > > > the
> > > > > > > > >> vector
> > > > > > > > >>
> > > > > > > > >> the first two arguments are quite weak by themselves. Vector
> > > > > > databases
> > > > > > > > are
> > > > > > > > >> very specialized engines. Such engines can of course be very
> > > > > > > > opinionated on
> > > > > > > > >> their data types. A general format like Parquet must instead
> > > > weigh
> > > > > > > > whether
> > > > > > > > >> a special type is worth the added complexity over a more
> > > > general,
> > > > > > > widely
> > > > > > > > >> applicable type. In this case, it is IMHO not at all. And
> > just
> > > > > > having
> > > > > > > a
> > > > > > > > >> type called "vector" for advertisement reasons to make
> > Parquet
> > > > > > appear
> > > > > > > > more
> > > > > > > > >> modern seems a non-goal in the first place.
> > > > > > > > >>
> > > > > > > > >> The third "time to market" argument is valid for systems
> > that
> > > > > don't
> > > > > > > > need a
> > > > > > > > >> stable interface. It is a very dangerous attitude for
> > formats
> > > > like
> > > > > > > > Parquet
> > > > > > > > >> that must be stable long term and should have a
> > consolidated,
> > > > > > concise,
> > > > > > > > >> non-redundant type system. For such long term storage
> > formats,
> > > > we
> > > > > > need
> > > > > > > > to
> > > > > > > > >> take the time to build the right thing, as any shortcuts we
> > > take
> > > > > > will
> > > > > > > > stay
> > > > > > > > >> as blemishes in the format forever. If I read the discussion
> > > > > > > correctly,
> > > > > > > > >> this is actually the main argument as of now: We have found
> > > that
> > > > > it
> > > > > > is
> > > > > > > > hard
> > > > > > > > >> to align on how a fixed size list type should look like, so
> > we
> > > > > want
> > > > > > to
> > > > > > > > go
> > > > > > > > >> the easier route of a specialized type. This seems like a
> > > > > > > > >> dangerous shortcut fueled by feature recency bias. Ask
> > > yourself:
> > > > > > Would
> > > > > > > > we
> > > > > > > > >> want this, if Parquet already had a fixed size list type?
> > > > > > > > >>
> > > > > > > > >> So finally, the only technical argument is the fourth one:
> > "It
> > > > > > signals
> > > > > > > > >> semantics and ensures that no bad data lands in the
> > format". I
> > > > see
> > > > > > two
> > > > > > > > big
> > > > > > > > >> problems with this:
> > > > > > > > >>
> > > > > > > > >> 1. Readers *still* need to test for NaN / INF because
> > Parquet
> > > > > files
> > > > > > > may
> > > > > > > > be
> > > > > > > > >> corrupt. Nothing can stop a malicious or buggy writer from
> > > > > > physically
> > > > > > > > >> writing NaN or INF. *So you save no single line of code in a
> > > > > reader.
> > > > > > > You
> > > > > > > > >> save no branches, as you still need to handle the INF/NaN
> > > case.*
> > > > > So
> > > > > > > this
> > > > > > > > >> doesn't help with performance at all. The only difference
> > is,
> > > > that
> > > > > > the
> > > > > > > > >> error message changes from "found an invalid INF value in
> > the
> > > > > data"
> > > > > > to
> > > > > > > > >> "Your parquet file seems to be corrupt". It is actually
> > quite
> > > > the
> > > > > > > > opposite,
> > > > > > > > >> readers and writers get more complex having to handle this
> > new
> > > > > type.
> > > > > > > > >>
> > > > > > > > >> If you now argue that it's not about performance, it's about
> > > > > defined
> > > > > > > > >> semantics and erroring out early in writers, then the second
> > > > > problem
> > > > > > > > >> arises:
> > > > > > > > >>
> > > > > > > > >> 2. The proposed no-NaN/INF/NULL guarantee is just one of
> > > > multiple
> > > > > > > > >> guarantees that you need for sensible vector processing. If
> > > "no
> > > > > > > > >> INF/NaN/NULL" was the only guarantee you needed, I would
> > agree
> > > > > that
> > > > > > > > >> FINITE_VECTOR is useful, but it is by no means the only
> > > > guarantee
> > > > > > you
> > > > > > > > need,
> > > > > > > > >> instead, depending on the distance metric, you also need the
> > > > > > following
> > > > > > > > >> guarantees. E.g.:
> > > > > > > > >>
> > > > > > > > >>   - No vector may be the 0 vector, otherwise, e.g., cosine
> > > > > > similarity
> > > > > > > is
> > > > > > > > >>   undefined. Cosine similarity is often used as default, so
> > > this
> > > > > is
> > > > > > > > >> actually
> > > > > > > > >>   quite important in practice.
> > > > > > > > >>   - If we want to use a dot product, then all vectors need
> > to
> > > > > > > guarantee
> > > > > > > > a
> > > > > > > > >>   length of 1. This would actually be quite useful for
> > > > performance
> > > > > > > > >> reasons.
> > > > > > > > >>   - The vectors must come from the same model as the vector
> > > used
> > > > > for
> > > > > > > the
> > > > > > > > >>   search. Otherwise, the search yields a numeric result, but
> > > > that
> > > > > > > result
> > > > > > > > >> has
> > > > > > > > >>   no sensible meaning.
> > > > > > > > >>
> > > > > > > > >> So, what use is a type that just guarantees part of the
> > things
> > > > you
> > > > > > > need
> > > > > > > > for
> > > > > > > > >> vector search but not others? You still need additional
> > > > guarantees
> > > > > > in
> > > > > > > > >> addition to that type. What makes INF/NaN so important in
> > > > > comparison
> > > > > > > to,
> > > > > > > > >> say, "no 0 vector present", that it warrants an own type,
> > > while
> > > > > the
> > > > > > > > other
> > > > > > > > >> guarantees do not? If we really want this type as a basis
> > for
> > > > > > > > cross-engine
> > > > > > > > >> sharing of vectors, then it needs to give *all* guarantees.
> > > > > > > > >>
> > > > > > > > >> I'm not against a specialized vector type if it really
> > carries
> > > > > *all*
> > > > > > > > >> information to be useful by itself. If it carried all that
> > > > > > > information,
> > > > > > > > >> then a reading engine could indeed validate that only
> > > meaningful
> > > > > > > vector
> > > > > > > > >> searches are executed. But as long as it is just a
> > half-baked
> > > > > thing,
> > > > > > > it
> > > > > > > > is
> > > > > > > > >> not useful by itself and just a burden on each implementer
> > and
> > > > > > > > maintainer.
> > > > > > > > >> If we need extra information anyway, why not just stick to a
> > > > > general
> > > > > > > > type
> > > > > > > > >> like VECTOR (without FINITE).
> > > > > > > > >>
> > > > > > > > >> If you're a proponent of FINITE_VECTOR, ask yourself (or
> > tell
> > > > us)
> > > > > > what
> > > > > > > > >> advantage you will *really* get over just a (potentially
> > fixed
> > > > > size)
> > > > > > > > List
> > > > > > > > >> of Float? As discussed, it's not performance. It's also not
> > a
> > > > > > semantic
> > > > > > > > >> marker that is sufficient by itself. It's just a small hint
> > > that
> > > > > > > allows
> > > > > > > > >> writers to error out once they see a NaN or INF in the
> > input,
> > > > > while
> > > > > > > they
> > > > > > > > >> will still happily write a bogus 0 vector. This seems to be
> > of
> > > > > zero
> > > > > > > > >> benefit.
> > > > > > > > >>
> > > > > > > > >> Anecdotally, we currently handle petabytes of vectors in
> > > > > production
> > > > > > in
> > > > > > > > >> Data360. We simply store them in a List of Float, with an
> > > > > optimized
> > > > > > > > R-Level
> > > > > > > > >> reader implementation that rapidly checks that all vectors
> > are
> > > > the
> > > > > > > same
> > > > > > > > >> length. This solution works just fine. The overhead of
> > R-level
> > > > > > > decoding
> > > > > > > > is
> > > > > > > > >> barely measurable. The check is super fast and R-levels are
> > > tiny
> > > > > in
> > > > > > > > >> comparison to the payload vectors, so also the added I/O is
> > > > > > > negligible.
> > > > > > > > >> This is also why I'm so firm in my sentiment: We are
> > running a
> > > > > > > > production
> > > > > > > > >> grade system processing trillions of vectors right now and
> > see
> > > > no
> > > > > > big
> > > > > > > > >> issues with the types that Parquet currently provides.
> > > > > > > > >>
> > > > > > > > >> As I don't just want to be a naysayer and point out problems
> > > > with
> > > > > > the
> > > > > > > > >> current design, here are the directions that I would propose
> > > > > indeed:
> > > > > > > > >>
> > > > > > > > >>   - a general fixed size list type. Don't do shortcuts just
> > > > > because
> > > > > > > the
> > > > > > > > >>   discussion is too tedious.
> > > > > > > > >>   - a vector type that indeed carries enough information to
> > be
> > > > > > really
> > > > > > > > >>   useful for vector search by itself. (e.g., model name /
> > > > version,
> > > > > > > > >> optional
> > > > > > > > >>   guarantees for no 0 vector / normalized vector length).
> > With
> > > > > such
> > > > > > a
> > > > > > > > >> type,
> > > > > > > > >>   you could really do well defined vector searches even if
> > the
> > > > > > writing
> > > > > > > > >> engine
> > > > > > > > >>   is completely different to the reading one.
> > > > > > > > >>   - Standardize quantizations. This will be a real game
> > > changer
> > > > > for
> > > > > > > > >>   performance.
> > > > > > > > >>
> > > > > > > > >> Cheers,
> > > > > > > > >> Jan
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >>> Am Mi., 14. Okt. 2026 um 03:36 Uhr schrieb Gang Wu <
> > > > > > ustcwg@gmail.com
> > > > > > > >:
> > > > > > > > >>>
> > > > > > > > >>> I think separating them into FINITE_VECTOR and
> > > > FIXED_LENGTH_LIST
> > > > > > is a
> > > > > > > > >> good
> > > > > > > > >>> idea, though I think renaming FINITE_VECTOR to VECTOR looks
> > > > more
> > > > > > > > natural.
> > > > > > > > >>> Should we move forward with proposing FINITE_VECTOR/VECTOR
> > > > first?
> > > > > > > > >>>
> > > > > > > > >>> On Tue, Oct 13, 2026 at 6:26 PM Alkis Evlogimenos via dev <
> > > > > > > > >>> dev@parquet.apache.org> wrote:
> > > > > > > > >>>
> > > > > > > > >>>> I suggest we go with VECTOR for the finite vector to keep
> > > the
> > > > > name
> > > > > > > the
> > > > > > > > >>> same
> > > > > > > > >>>> as other engines/type systems. If we want a non-finite one
> > > in
> > > > > the
> > > > > > > > >> future
> > > > > > > > >>> we
> > > > > > > > >>>> can name that something else or we add fixed size list
> > that
> > > > will
> > > > > > > serve
> > > > > > > > >>> the
> > > > > > > > >>>> same purpose without a special name.
> > > > > > > > >>>>
> > > > > > > > >>>>
> > > > > > > > >>>> On Thu, Oct 8, 2026 at 10:20 PM Ryan Blue <
> > blue@apache.org>
> > > > > > wrote:
> > > > > > > > >>>>
> > > > > > > > >>>>> A logical type would:
> > > > > > > > >>>>> - Describe data for users and readers to understand what
> > > the
> > > > > file
> > > > > > > > >>>> contains
> > > > > > > > >>>>> - Signal that repetition and definition levels will
> > > reflect a
> > > > > > fixed
> > > > > > > > >>>> number
> > > > > > > > >>>>> of elements, which implementations can use to optimize
> > > > decoding
> > > > > > > > >>>>> - Guarantee that there are no null elements (because the
> > > type
> > > > > > > cannot
> > > > > > > > >> be
> > > > > > > > >>>>> applied to a list with optional elements)
> > > > > > > > >>>>> - [FINITE_VECTOR] Guarantee that no elements are NaN or
> > > > > Infinity
> > > > > > > > >> values
> > > > > > > > >>>> so
> > > > > > > > >>>>> that the vectors have a finite length and are suitable
> > for
> > > > > > distance
> > > > > > > > >>>>> calculations
> > > > > > > > >>>>>
> > > > > > > > >>>>> The finite value restriction is a restriction on writers
> > to
> > > > > > prevent
> > > > > > > > >> bad
> > > > > > > > >>>>> data from being written.
> > > > > > > > >>>>>
> > > > > > > > >>>>> Ignoring logical types that are not understood is a rule
> > > for
> > > > > > > > >> readers. A
> > > > > > > > >>>>> reader can handle vectors and pass it on to another
> > layer.
> > > > For
> > > > > > > > >>> instance,
> > > > > > > > >>>> a
> > > > > > > > >>>>> Parquet reader that doesn't support geometry passes
> > values
> > > as
> > > > > > > > >> `binary`.
> > > > > > > > >>>>> That binary value can be passed to a geometry UDF, even
> > if
> > > > the
> > > > > > > > >>> underlying
> > > > > > > > >>>>> engine and Parquet reader doesn't have support.
> > > > > > > > >>>>>
> > > > > > > > >>>>> A tool that doesn't understand FINITE_VECTOR can
> > similarly
> > > > > pass a
> > > > > > > > >> list
> > > > > > > > >>> of
> > > > > > > > >>>>> floats along to a UDF. But it is still important to the
> > > user
> > > > > that
> > > > > > > bad
> > > > > > > > >>>>> values are not present in the data. In your example, bad
> > > > values
> > > > > > are
> > > > > > > > >>>> vectors
> > > > > > > > >>>>> that cause the centroid calculation to fail, like vectors
> > > > with
> > > > > > > > >> infinite
> > > > > > > > >>>>> length.
> > > > > > > > >>>>>
> > > > > > > > >>>>> Ryan
> > > > > > > > >>>>>
> > > > > > > > >>>>> On Thu, Oct 8, 2026 at 12:54 PM Will Edwards via dev <
> > > > > > > > >>>>> dev@parquet.apache.org>
> > > > > > > > >>>>> wrote:
> > > > > > > > >>>>>
> > > > > > > > >>>>>> Could we clarify the use cases for a VECTOR annotation
> > on
> > > > > > > > >>> LIST<FLOAT>?
> > > > > > > > >>>>>> Specifically, what does it make possible that's
> > impossible
> > > > > > today,
> > > > > > > > >> and
> > > > > > > > >>>>> what
> > > > > > > > >>>>>> does it make easier that's currently hard?
> > > > > > > > >>>>>>
> > > > > > > > >>>>>> I'm struggling to find a strong answer. I'd like every
> > > > engine
> > > > > to
> > > > > > > > >>> offer
> > > > > > > > >>>>>> vector functions on lists of floats without needing a
> > new
> > > > > > logical
> > > > > > > > >>> type.
> > > > > > > > >>>>>>
> > > > > > > > >>>>>> To be clear, I'm still enthusiastic about a vector type
> > > that
> > > > > > > > >>> describes
> > > > > > > > >>>>> how
> > > > > > > > >>>>>> to unpack and scale composite representations (quantised
> > > > codes
> > > > > > > plus
> > > > > > > > >>>>> scales,
> > > > > > > > >>>>>> codebooks, etc.), in the direction ML is going. That
> > adds
> > > > > > > > >> information
> > > > > > > > >>>>>> engines don't have. A label on LIST<FLOAT> doesn't:
> > > engines
> > > > > > > already
> > > > > > > > >>>>>> understand the physical type, and many already offer
> > > vector
> > > > > > > > >> functions
> > > > > > > > >>>> on
> > > > > > > > >>>>>> it. I don't see a performance benefit, and NaN/null/Inf
> > > > checks
> > > > > > are
> > > > > > > > >>>> cheap
> > > > > > > > >>>>>> enough that engines should do them anyway. The label
> > also
> > > > > > doesn't
> > > > > > > > >> say
> > > > > > > > >>>>> what
> > > > > > > > >>>>>> you can do with the column. Knowing that a column holds
> > > > > > embeddings
> > > > > > > > >>>>> doesn't
> > > > > > > > >>>>>> say how they can be used, just as zipcode and age are
> > both
> > > > > > > integers
> > > > > > > > >>> but
> > > > > > > > >>>>>> multiplying them is meaningless.
> > > > > > > > >>>>>>
> > > > > > > > >>>>>> My concern is the opposite effect. We're establishing
> > that
> > > > > > engines
> > > > > > > > >>>> should
> > > > > > > > >>>>>> ignore logical types they don't understand. So a tool
> > that
> > > > > > doesn't
> > > > > > > > >>> know
> > > > > > > > >>>>>> VECTOR will read the column as LIST<FLOAT>, let the user
> > > > > compute
> > > > > > > > >>>>> centroids
> > > > > > > > >>>>>> (say), and write plain LIST<FLOAT> back out. If other
> > > > engines
> > > > > > > > >>> restrict
> > > > > > > > >>>>>> embedding functions to VECTOR-annotated columns, that
> > file
> > > > > stops
> > > > > > > > >>>> working
> > > > > > > > >>>>>> with them, through no fault of the user. An annotation
> > > that
> > > > > > > doesn't
> > > > > > > > >>>>> change
> > > > > > > > >>>>>> what the data is shouldn't be able to make it unusable.
> > > > > > > > >>>>>>
> > > > > > > > >>>>>> Am I missing a use case?
> > > > > > > > >>>>>>
> > > > > > > > >>>>>> Best, Will
> > > > > > > > >>>>>>
> > > > > > > > >>>>>> On Thu, 8 Oct 2026 at 19:53, Steven Wu <
> > > > stevenz3wu@gmail.com>
> > > > > > > > >> wrote:
> > > > > > > > >>>>>>
> > > > > > > > >>>>>>> I also think a FINITE_VECTOR type brings a lot of
> > values
> > > to
> > > > > > many
> > > > > > > > >>> use
> > > > > > > > >>>>>> cases.
> > > > > > > > >>>>>>> It is would be great to make progress on the
> > > FINITE_VECTOR,
> > > > > and
> > > > > > > > >> not
> > > > > > > > >>>> to
> > > > > > > > >>>>>> fit
> > > > > > > > >>>>>>> every use cases into the FIXED_LENGTH_LIST.
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>> On Mon, Oct 5, 2026 at 1:26 PM Ryan Blue <
> > > blue@apache.org>
> > > > > > > > >> wrote:
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>>> Going through this thread again, it looks like there
> > > are a
> > > > > few
> > > > > > > > >>>> things
> > > > > > > > >>>>>>> that
> > > > > > > > >>>>>>>> people generally agree on:
> > > > > > > > >>>>>>>>
> > > > > > > > >>>>>>>> - In the short term, we want to introduce a logical
> > type
> > > > or
> > > > > > > > >>> types;
> > > > > > > > >>>> a
> > > > > > > > >>>>>>> better
> > > > > > > > >>>>>>>> physical representation can be done later. We can
> > focus
> > > on
> > > > > > > > >>> logical
> > > > > > > > >>>>>> types.
> > > > > > > > >>>>>>>> - There is precedent in logical types for restricting
> > > > > values,
> > > > > > > > >>> like
> > > > > > > > >>>>> UTF8
> > > > > > > > >>>>>>>> strings to contain only UTF-8 codepoints. I think we
> > > also
> > > > > > agree
> > > > > > > > >>>> that
> > > > > > > > >>>>> we
> > > > > > > > >>>>>>>> could restrict vector elements to non-nested or
> > > primitive
> > > > > > > > >> types.
> > > > > > > > >>>>>>>> - There are many examples of databases that restrict
> > > their
> > > > > > > > >> vector
> > > > > > > > >>>>> types
> > > > > > > > >>>>>>> to
> > > > > > > > >>>>>>>> non-null, finite elements.
> > > > > > > > >>>>>>>> - Some people want to have write-time guarantees that
> > > > > vectors
> > > > > > > > >>> have
> > > > > > > > >>>>> only
> > > > > > > > >>>>>>>> non-null, finite elements
> > > > > > > > >>>>>>>> - Some people want to extend this work to include
> > > tensors,
> > > > > > > > >>> shorter
> > > > > > > > >>>>>>> floating
> > > > > > > > >>>>>>>> point representations, variable lengths, NaN or
> > Infinity
> > > > > > > > >>> elements,
> > > > > > > > >>>>> etc.
> > > > > > > > >>>>>>>>
> > > > > > > > >>>>>>>> It sounds like we are nearly there, but have two
> > halves
> > > of
> > > > > the
> > > > > > > > >>>>>> community
> > > > > > > > >>>>>>>> that care about different things. That just means we
> > > > should
> > > > > > > > >> have
> > > > > > > > >>>>>> multiple
> > > > > > > > >>>>>>>> types. It's entirely reasonable to want a type that
> > > allows
> > > > > NaN
> > > > > > > > >>>>> values,
> > > > > > > > >>>>>>> and
> > > > > > > > >>>>>>>> it's also reasonable for those of us building
> > databases,
> > > > > table
> > > > > > > > >>>>>> standards,
> > > > > > > > >>>>>>>> and vector indexes to care about guarantees that
> > > elements
> > > > > are
> > > > > > > > >>>> finite.
> > > > > > > > >>>>>>>> There's more than enough examples to show that this
> > is a
> > > > > > > > >>> legitimate
> > > > > > > > >>>>> use
> > > > > > > > >>>>>>>> case.
> > > > > > > > >>>>>>>>
> > > > > > > > >>>>>>>> Since I don't see the concerns of the different use
> > > cases
> > > > > > > > >>> changing,
> > > > > > > > >>>>> it
> > > > > > > > >>>>>>>> looks like we need to find a compromise to move
> > > forward. I
> > > > > > > > >>> propose
> > > > > > > > >>>>> that
> > > > > > > > >>>>>>> we
> > > > > > > > >>>>>>>> introduce a FINITE_VECTOR type and separately a
> > > > > > > > >> FIXED_LENGTH_LIST
> > > > > > > > >>>> or
> > > > > > > > >>>>>>>> another VECTOR type that allows infinite and NaN
> > values.
> > > > > > > > >>>>>>>>
> > > > > > > > >>>>>>>> Can we agree on a compromise? Is there a strong reason
> > > to
> > > > > try
> > > > > > > > >> to
> > > > > > > > >>>> fit
> > > > > > > > >>>>>>> these
> > > > > > > > >>>>>>>> use cases into a single type? What if that means not
> > > > making
> > > > > > > > >>>> progress?
> > > > > > > > >>>>>>>>
> > > > > > > > >>>>>>>> Ryan
> > > > > > > > >>>>>>>>
> > > > > > > > >>>>>>>> On Mon, Oct 5, 2026 at 12:15 PM Philipp Fischbeck via
> > > dev
> > > > <
> > > > > > > > >>>>>>>> dev@parquet.apache.org> wrote:
> > > > > > > > >>>>>>>>
> > > > > > > > >>>>>>>>> Thinking about this more, I am also fine with pulling
> > > out
> > > > > the
> > > > > > > > >>>>>> “NaN/Inf”
> > > > > > > > >>>>>>>>> requirement. Either into a type boolean (my
> > > preference),
> > > > or
> > > > > > > > >>>> putting
> > > > > > > > >>>>>> no
> > > > > > > > >>>>>>>>> restrictions on the Parquet level, allowing all IEEE
> > > 754
> > > > > > > > >> float
> > > > > > > > >>>>>> numbers.
> > > > > > > > >>>>>>>> If
> > > > > > > > >>>>>>>>> some top-level application wants to put additional
> > > > > > > > >> constraints
> > > > > > > > >>> on
> > > > > > > > >>>>> the
> > > > > > > > >>>>>>>> data
> > > > > > > > >>>>>>>>> type, like Iceberg not allowing NaN/infinity, it can
> > > > still
> > > > > do
> > > > > > > > >>> so
> > > > > > > > >>>>> one
> > > > > > > > >>>>>>>> layer
> > > > > > > > >>>>>>>>> above. It seems like this was the biggest point of
> > > > > discussion
> > > > > > > > >>>>>>> remaining.
> > > > > > > > >>>>>>>>> All these options work for me, and we should move
> > > forward
> > > > > > > > >> with
> > > > > > > > >>>> this
> > > > > > > > >>>>>>>>> proposal.
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>>> Concerning additional element types like bf16, they
> > can
> > > > be
> > > > > > > > >>>>> introduced
> > > > > > > > >>>>>>> as
> > > > > > > > >>>>>>>>> Parquet types (at least in the current LIST-based
> > spec
> > > > > > > > >>> proposal).
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>>> Concerning Will’s suggestion around representing
> > VECTOR
> > > > as
> > > > > a
> > > > > > > > >>>>> variable
> > > > > > > > >>>>>>>>> length list with a MIN/MAX_LENGTH metadata
> > > restriction: I
> > > > > > > > >> think
> > > > > > > > >>>>> that
> > > > > > > > >>>>>> a
> > > > > > > > >>>>>>>>> dedicated logical VECTOR type still provides enough
> > > > > benefits
> > > > > > > > >> by
> > > > > > > > >>>>>>>> describing
> > > > > > > > >>>>>>>>> the semantics of the data, and allowing to capture
> > > future
> > > > > > > > >>> changes
> > > > > > > > >>>>>>> around
> > > > > > > > >>>>>>>>> statistics. In that case, the length is better
> > captured
> > > > as
> > > > > an
> > > > > > > > >>>> exact
> > > > > > > > >>>>>>>> length
> > > > > > > > >>>>>>>>> for this type specifically. I’m also not sure about
> > > other
> > > > > use
> > > > > > > > >>>> cases
> > > > > > > > >>>>>> for
> > > > > > > > >>>>>>>>> MIN_LENGTH, so introducing a new type constraint
> > system
> > > > > just
> > > > > > > > >>> for
> > > > > > > > >>>>> the
> > > > > > > > >>>>>>>>> MIN_LENGTH=MAX_LENGTH case for vectors doesn’t have
> > > > enough
> > > > > > > > >>> value
> > > > > > > > >>>> in
> > > > > > > > >>>>>> my
> > > > > > > > >>>>>>>>> view.
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>>> Best,
> > > > > > > > >>>>>>>>> Philipp
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>>> On Thu, Oct 1, 2026 at 9:00 AM Will Edwards via dev <
> > > > > > > > >>>>>>>>> dev@parquet.apache.org>
> > > > > > > > >>>>>>>>> wrote:
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>>>> +1 Sebastian, a vector type suggests usage beyond
> > > > > > > > >> normalized
> > > > > > > > >>>>>>> embeddings
> > > > > > > > >>>>>>>>> and
> > > > > > > > >>>>>>>>>> engines really aren't going to be able to elide all
> > > the
> > > > > > > > >>> checks
> > > > > > > > >>>>>>> anyway.
> > > > > > > > >>>>>>>>>>
> > > > > > > > >>>>>>>>>> Antoine, you suggested we explore metadata solutions
> > > too
> > > > > :)
> > > > > > > > >>>>>>>>>>
> > > > > > > > >>>>>>>>>> Today, a database supporting CHAR(N) or VARCHAR(n)
> > > can't
> > > > > > > > >>>> express
> > > > > > > > >>>>>> this
> > > > > > > > >>>>>>>> in
> > > > > > > > >>>>>>>>>> Parquet.  Engines can write opaque blobs they
> > > understand
> > > > > in
> > > > > > > > >>> the
> > > > > > > > >>>>>>>> metadata
> > > > > > > > >>>>>>>>>> (e.g. Arrow does) so the extra type info
> > round-trips,
> > > > but
> > > > > > > > >>> this
> > > > > > > > >>>>>> isn't
> > > > > > > > >>>>>>>>>> standardized, portable, nor interoperable.
> > > > > > > > >>>>>>>>>>
> > > > > > > > >>>>>>>>>> So perhaps standardizing constraint metadata or new
> > > > Thrift
> > > > > > > > >>>> fields
> > > > > > > > >>>>>> (my
> > > > > > > > >>>>>>>>> pref)
> > > > > > > > >>>>>>>>>> can generically support VECTOR(N) whilst also being
> > > more
> > > > > > > > >>>> broadly
> > > > > > > > >>>>>>> useful
> > > > > > > > >>>>>>>>> for
> > > > > > > > >>>>>>>>>> other use cases and data types?
> > > > > > > > >>>>>>>>>>
> > > > > > > > >>>>>>>>>> Imagine metadata or Thrift fields to tag the
> > > MIN_LENGTH
> > > > > and
> > > > > > > > >>>>>>> MAX_LENGTH
> > > > > > > > >>>>>>>>> for
> > > > > > > > >>>>>>>>>> Parquet's variable-length physcial types.  These
> > would
> > > > > > > > >> count
> > > > > > > > >>>>>> logical
> > > > > > > > >>>>>>>>> units
> > > > > > > > >>>>>>>>>> (e.g. UTF8 code points for STRING) rather than
> > bytes.
> > > > > > > > >>>>>>>>>>
> > > > > > > > >>>>>>>>>> Then one engine's VECTOR(N) can be written to
> > Parquet
> > > as
> > > > > > > > >>>> LIST(N)
> > > > > > > > >>>>>> with
> > > > > > > > >>>>>>>>>> MIN_LENGTH = MAX_LENGTH = N and then readers - even
> > > > > > > > >> different
> > > > > > > > >>>>>>> engines -
> > > > > > > > >>>>>>>>> can
> > > > > > > > >>>>>>>>>> pick up these constraints if they support doing so.
> > > We
> > > > > > > > >> don't
> > > > > > > > >>>>>>> actually
> > > > > > > > >>>>>>>>> need
> > > > > > > > >>>>>>>>>> a logical type to express a VECTOR then.
> > > > > > > > >>>>>>>>>>
> > > > > > > > >>>>>>>>>> This also allows standard SQL's CHAR(N) and
> > VARCHAR(N)
> > > > to
> > > > > > > > >> be
> > > > > > > > >>>>>>> expressed
> > > > > > > > >>>>>>>> on
> > > > > > > > >>>>>>>>>> UTF8 columns etc.
> > > > > > > > >>>>>>>>>>
> > > > > > > > >>>>>>>>>> This also seems generally useful in the future as we
> > > > > > > > >> approach
> > > > > > > > >>>>>>>> expressing
> > > > > > > > >>>>>>>>>> other tensor data shapes and types.
> > > > > > > > >>>>>>>>>>
> > > > > > > > >>>>>>>>>> This won't change the mechanics: the storage type
> > > > doesn't
> > > > > > > > >>>> enforce
> > > > > > > > >>>>>>> these
> > > > > > > > >>>>>>>>>> constraints and engines must check them. It's
> > > previously
> > > > > > > > >>>>>> established
> > > > > > > > >>>>>>>> that
> > > > > > > > >>>>>>>>>> performance is just a software problem and there is
> > no
> > > > > > > > >>>>> fundamental
> > > > > > > > >>>>>>>>>> performance advantage to the fixed size types in
> > > > > > > > >> Dremel-like
> > > > > > > > >>>>>> formats
> > > > > > > > >>>>>>>> like
> > > > > > > > >>>>>>>>>> Parquet.
> > > > > > > > >>>>>>>>>>
> > > > > > > > >>>>>>>>>> Best,
> > > > > > > > >>>>>>>>>> Will
> > > > > > > > >>>>>>>>>>
> > > > > > > > >>>>>>>>>> On Wed, 30 Sept 2026 at 16:08, Sebastian Baunsgaard
> > <
> > > > > > > > >>>>>>>>> baunsgaard@apache.org
> > > > > > > > >>>>>>>>>>>
> > > > > > > > >>>>>>>>>> wrote:
> > > > > > > > >>>>>>>>>>
> > > > > > > > >>>>>>>>>>> After listening in on the last meeting Wednesday, I
> > > > have
> > > > > > > > >>> been
> > > > > > > > >>>>>>>> thinking
> > > > > > > > >>>>>>>>>>> about this quite a bit, and thought I would share
> > my
> > > 2
> > > > > > > > >>> cents.
> > > > > > > > >>>>>>>>>>>
> > > > > > > > >>>>>>>>>>> I suggest not requiring VECTOR elements to be
> > finite,
> > > > as
> > > > > > > > >>>>> proposed
> > > > > > > > >>>>>>> in
> > > > > > > > >>>>>>>>> the
> > > > > > > > >>>>>>>>>> PR
> > > > > > > > >>>>>>>>>>> <https://github.com/apache/parquet-format/pull/624
> > >.
> > > > > > > > >>>>>>>>>>>
> > > > > > > > >>>>>>>>>>> While Divjot and others note the specialization for
> > > > > > > > >> UTF-8,
> > > > > > > > >>>>>> Parquet
> > > > > > > > >>>>>>>>>> already
> > > > > > > > >>>>>>>>>>> defines DOUBLE and FLOAT as IEEE encodings
> > > > > > > > >>>>>>>>>>> <
> > > > > > > > >>>>>>>>
> > > > > > > > >>>>
> > > > > > >
> > > https://github.com/apache/parquet-format/blob/master/README.md#types
> > > > > > > > >>>>>>>>>> .
> > > > > > > > >>>>>>>>>>>
> > > > > > > > >>>>>>>>>>> Making NaN or +/-Inf illegal inside VECTOR would
> > be a
> > > > new
> > > > > > > > >>>>>>> restriction
> > > > > > > > >>>>>>>>> on
> > > > > > > > >>>>>>>>>>> those same underlying physical types. Scalar
> > > > FLOAT/DOUBLE
> > > > > > > > >>>>> columns
> > > > > > > > >>>>>>>> have
> > > > > > > > >>>>>>>>>>> never had such a rule.
> > > > > > > > >>>>>>>>>>>
> > > > > > > > >>>>>>>>>>> It would also not match how the systems that
> > actually
> > > > > > > > >>> produce
> > > > > > > > >>>>>> these
> > > > > > > > >>>>>>>>>> vectors
> > > > > > > > >>>>>>>>>>> treat floats. Most of them use IEEE floats, e.g.
> > > Spark,
> > > > > > > > >>>> DuckDB,
> > > > > > > > >>>>>>>> NumPy,
> > > > > > > > >>>>>>>>>>> PyTorch, TensorFlow, or lower level CUDA, oneMKL,
> > and
> > > > > > > > >>>> OpenBLAS.
> > > > > > > > >>>>>>>>>>>
> > > > > > > > >>>>>>>>>>> pgvector's "all elements must be finite" rule is an
> > > > > > > > >> insert
> > > > > > > > >>>>>> policy.
> > > > > > > > >>>>>>>>> While
> > > > > > > > >>>>>>>>>> I
> > > > > > > > >>>>>>>>>>> think Parquet as a general-purpose format should
> > > allow
> > > > > > > > >> them
> > > > > > > > >>>> out
> > > > > > > > >>>>>> of
> > > > > > > > >>>>>>>> the
> > > > > > > > >>>>>>>>>> box.
> > > > > > > > >>>>>>>>>>> It is a user system decision to disallow writing
> > > these
> > > > > > > > >>> values
> > > > > > > > >>>>> or
> > > > > > > > >>>>>> to
> > > > > > > > >>>>>>>>> throw
> > > > > > > > >>>>>>>>>>> errors when encountering them.
> > > > > > > > >>>>>>>>>>>
> > > > > > > > >>>>>>>>>>> A type-level ban also does not keep Inf/NaN out of
> > > > vector
> > > > > > > > >>>> math.
> > > > > > > > >>>>>>>> Finite
> > > > > > > > >>>>>>>>>>> coordinates still overflow. For instance:
> > magnitude =
> > > > > > > > >>>>>> ||[FLOAT_MAX,
> > > > > > > > >>>>>>>>>>> FLOAT_MAX]|| where every coordinate is finite
> > > overflows
> > > > > > > > >> to
> > > > > > > > >>>>> +Inf.
> > > > > > > > >>>>>>>>> PyTorch
> > > > > > > > >>>>>>>>>>> documents the same overflow: tensor([1e20,
> > > > 1e20]).norm()
> > > > > > > > >> is
> > > > > > > > >>>> inf
> > > > > > > > >>>>>> in
> > > > > > > > >>>>>>>>> fp32.
> > > > > > > > >>>>>>>>>>>
> > > > > > > > >>>>>>>>>>> A generated MAGNITUDE(vec) column would then be Inf
> > > > while
> > > > > > > > >>> the
> > > > > > > > >>>>>>> VECTOR
> > > > > > > > >>>>>>>>>> itself
> > > > > > > > >>>>>>>>>>> was required to be finite. The ban only forbids
> > > storing
> > > > > > > > >>>>> specials
> > > > > > > > >>>>>> in
> > > > > > > > >>>>>>>> the
> > > > > > > > >>>>>>>>>>> coordinates, and does not prevent even simple
> > > > downstream
> > > > > > > > >>>>>>> computations
> > > > > > > > >>>>>>>>>> from
> > > > > > > > >>>>>>>>>>> producing them.
> > > > > > > > >>>>>>>>>>>
> > > > > > > > >>>>>>>>>>> I suggest keeping VECTOR as the type and add an
> > > > optional
> > > > > > > > >>> bool
> > > > > > > > >>>>>>>>>>> VectorType.finite (default false). Engines that
> > want
> > > > > > > > >>>>>> pgvector-like
> > > > > > > > >>>>>>>>>>> rejection can require the flag = true. Engines that
> > > use
> > > > > > > > >>>> NumPy /
> > > > > > > > >>>>>>>>> PyTorch /
> > > > > > > > >>>>>>>>>>> Spark arrays can keep representable IEEE values.
> > For
> > > > the
> > > > > > > > >>> same
> > > > > > > > >>>>>>>> reason, I
> > > > > > > > >>>>>>>>>> do
> > > > > > > > >>>>>>>>>>> not favor adding yet another type called
> > > FINITE_VECTOR.
> > > > > > > > >>>>>>>>>>>
> > > > > > > > >>>>>>>>>>> Best,
> > > > > > > > >>>>>>>>>>> Sebastian
> > > > > > > > >>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>
> > > > > > > > >>>>>>>>>>> On Tue, Sep 29, 2026 at 2:03 PM Divjot Arora via
> > dev
> > > <
> > > > > > > > >>>>>>>>>>> dev@parquet.apache.org>
> > > > > > > > >>>>>>>>>>> wrote:
> > > > > > > > >>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>> I agree with moving forward with the proposal and
> > > > > > > > >>> renaming
> > > > > > > > >>>>>> VECTOR
> > > > > > > > >>>>>>>> to
> > > > > > > > >>>>>>>>>>>> FINITE_VECTOR or something
> > > > > > > > >>>>>>>>>>>> similar if others feel that better communicates
> > its
> > > > > > > > >>>> intent. I
> > > > > > > > >>>>>> see
> > > > > > > > >>>>>>>>> value
> > > > > > > > >>>>>>>>>>> in
> > > > > > > > >>>>>>>>>>>> having a vector type that guarantees
> > > > > > > > >>>>>>>>>>>> its values are mathematically-correct vectors for
> > a
> > > > few
> > > > > > > > >>>>>> reasons:
> > > > > > > > >>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>> 1. As others have mentioned, this is similar to
> > the
> > > > > > > > >>> STRING
> > > > > > > > >>>>>>> logical
> > > > > > > > >>>>>>>>> type
> > > > > > > > >>>>>>>>>>>> mandating UTF-8.
> > > > > > > > >>>>>>>>>>>> 2. It is not possible to rely on statistics for
> > > > > > > > >> detecting
> > > > > > > > >>>> NaN
> > > > > > > > >>>>>> or
> > > > > > > > >>>>>>>>> +-inf
> > > > > > > > >>>>>>>>>>>> values as all statistics are optional.
> > > > > > > > >>>>>>>>>>>> To my knowledge, there is no precedent for
> > requiring
> > > > > > > > >> that
> > > > > > > > >>>>>> writers
> > > > > > > > >>>>>>>>> emit
> > > > > > > > >>>>>>>>>>>> stats for specific logical types.
> > > > > > > > >>>>>>>>>>>> 3. Without statistics, NaN/inf values could flow
> > > > > > > > >> through
> > > > > > > > >>>> the
> > > > > > > > >>>>>>> engine
> > > > > > > > >>>>>>>>> and
> > > > > > > > >>>>>>>>>>>> block them for specific functionality.
> > > > > > > > >>>>>>>>>>>> This seems to be Lance's behavior. However, I
> > don't
> > > > > > > > >> think
> > > > > > > > >>>>> this
> > > > > > > > >>>>>> is
> > > > > > > > >>>>>>>> as
> > > > > > > > >>>>>>>>>> easy
> > > > > > > > >>>>>>>>>>>> as it sounds and Lance has run
> > > > > > > > >>>>>>>>>>>> into several bugs due to this ([1] [2] for
> > > examples).
> > > > > > > > >>> Their
> > > > > > > > >>>>>>> Python
> > > > > > > > >>>>>>>>> [3]
> > > > > > > > >>>>>>>>>>> and
> > > > > > > > >>>>>>>>>>>> Rust [4] APIs default to erroring on
> > > > > > > > >>>>>>>>>>>> vectors that contain NaN values.
> > > > > > > > >>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>> I agree that Parquet is a general-purpose format,
> > > but
> > > > I
> > > > > > > > >>>> don't
> > > > > > > > >>>>>>> think
> > > > > > > > >>>>>>>>>> this
> > > > > > > > >>>>>>>>>>>> implies the lack of domain-specific types.
> > > > > > > > >>>>>>>>>>>> Instead, this broadens the use cases where Parquet
> > > > > > > > >> would
> > > > > > > > >>> be
> > > > > > > > >>>>>>>>> considered
> > > > > > > > >>>>>>>>>> a
> > > > > > > > >>>>>>>>>>>> good fit. I also fully agree with
> > > > > > > > >>>>>>>>>>>> continuing work on a general fixed-size list type
> > if
> > > > we
> > > > > > > > >>>>> believe
> > > > > > > > >>>>>>>> there
> > > > > > > > >>>>>>>>>> are
> > > > > > > > >>>>>>>>>>>> non-vector use cases that would benefit
> > > > > > > > >>>>>>>>>>>> from such guarantees at the storage layer.
> > > > > > > > >>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>> [1]
> > https://github.com/lance-format/lance/pull/3648
> > > > > > > > >>>>>>>>>>>> [2]
> > > https://github.com/lance-format/lance/issues/8607
> > > > > > > > >>> (see
> > > > > > > > >>>>>>> point 2
> > > > > > > > >>>>>>>>> in
> > > > > > > > >>>>>>>>>>>> "root
> > > > > > > > >>>>>>>>>>>> cause")
> > > > > > > > >>>>>>>>>>>> [3]
> > > > > > > > >>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>
> > > > > > > > >>>>>>>>>>
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>>
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>
> > > > > > > > >>>>>
> > > > > > > > >>>>
> > > > > > > > >>>
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > https://github.com/lancedb/lancedb/blob/0327618a62483feb6dca2336549845e5887cd48f/python/python/lancedb/table.py#L4807-L4911
> > > > > > > > >>>>>>>>>>>> [4]
> > > > > > > > >>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>
> > > > > > > > >>>>>>>>>>
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>>
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>
> > > > > > > > >>>>>
> > > > > > > > >>>>
> > > > > > > > >>>
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > https://github.com/lancedb/lancedb/blob/0327618a62483feb6dca2336549845e5887cd48f/rust/lancedb/src/table/add_data.rs#L105-L110
> > > > > > > > >>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>> On Mon, Sep 28, 2026 at 6:19 PM Philipp Fischbeck
> > > via
> > > > > > > > >>> dev <
> > > > > > > > >>>>>>>>>>>> dev@parquet.apache.org> wrote:
> > > > > > > > >>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>> I also favor moving forward with the proposal as
> > > is.
> > > > > > > > >> I
> > > > > > > > >>>> see
> > > > > > > > >>>>>>> strong
> > > > > > > > >>>>>>>>>> value
> > > > > > > > >>>>>>>>>>>> and
> > > > > > > > >>>>>>>>>>>>> use cases for the numeric vector type, and I
> > agree
> > > > > > > > >> with
> > > > > > > > >>>> the
> > > > > > > > >>>>>>>>>>> restrictions
> > > > > > > > >>>>>>>>>>>>> around NaN/Inf values and element types in
> > general.
> > > > > > > > >>>>> Precedent
> > > > > > > > >>>>>>>>> exists
> > > > > > > > >>>>>>>>>> in
> > > > > > > > >>>>>>>>>>>>> other types that restrict which physical type
> > > values
> > > > > > > > >>> are
> > > > > > > > >>>>>>>> logically
> > > > > > > > >>>>>>>>>>> valid
> > > > > > > > >>>>>>>>>>>>> (e.g., the geospatial types needing to be
> > > > > > > > >> WKB-encoded).
> > > > > > > > >>>> A
> > > > > > > > >>>>>>>> general
> > > > > > > > >>>>>>>>>>>>> fixed-size list also has value, but I think it
> > > makes
> > > > > > > > >>>> sense
> > > > > > > > >>>>> to
> > > > > > > > >>>>>>>>>> separate
> > > > > > > > >>>>>>>>>>>> the
> > > > > > > > >>>>>>>>>>>>> two types. The semantics are relevant, and I can
> > > see
> > > > > > > > >>> how
> > > > > > > > >>>>> the
> > > > > > > > >>>>>>> two
> > > > > > > > >>>>>>>>>> types
> > > > > > > > >>>>>>>>>>>>> might diverge further in the future regarding
> > > > > > > > >>> statistics
> > > > > > > > >>>> or
> > > > > > > > >>>>>>>>>> encodings.
> > > > > > > > >>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>> I would be okay with renaming the vector type,
> > but
> > > it
> > > > > > > > >>>> seems
> > > > > > > > >>>>>>> like
> > > > > > > > >>>>>>>>> the
> > > > > > > > >>>>>>>>>>>> VECTOR
> > > > > > > > >>>>>>>>>>>>> name is already used across other engines
> > mentioned
> > > > > > > > >> in
> > > > > > > > >>>> this
> > > > > > > > >>>>>>>> thread.
> > > > > > > > >>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>> Best,
> > > > > > > > >>>>>>>>>>>>> Philipp
> > > > > > > > >>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>> On Sat, Sep 26, 2026 at 4:37 PM Gang Wu <
> > > > > > > > >>>> ustcwg@gmail.com>
> > > > > > > > >>>>>>>> wrote:
> > > > > > > > >>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>> To separate orthogonal concerns, I feel we
> > should
> > > > > > > > >>> treat
> > > > > > > > >>>>> the
> > > > > > > > >>>>>>>>> general
> > > > > > > > >>>>>>>>>>>>>> FixedSizeList as the underlying physical layout,
> > > > > > > > >> and
> > > > > > > > >>>>>> separate
> > > > > > > > >>>>>>>> it
> > > > > > > > >>>>>>>>>> from
> > > > > > > > >>>>>>>>>>>> the
> > > > > > > > >>>>>>>>>>>>>> discussion of a specialized VECTOR (or broader
> > ML
> > > > > > > > >>>> tensor)
> > > > > > > > >>>>>>>> logical
> > > > > > > > >>>>>>>>>>> type.
> > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>> Currently, the immediate need is to define a
> > > > > > > > >>>> specialized
> > > > > > > > >>>>>>> VECTOR
> > > > > > > > >>>>>>>>>> type
> > > > > > > > >>>>>>>>>>> so
> > > > > > > > >>>>>>>>>>>>>> that different engines and formats have a common
> > > > > > > > >>>> semantic
> > > > > > > > >>>>>>>>> contract
> > > > > > > > >>>>>>>>>>> for
> > > > > > > > >>>>>>>>>>>>>> interoperability.
> > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>> If the main point of contention is just the
> > > > > > > > >>>> NaN/Inf/null
> > > > > > > > >>>>>>>>>> limitations,
> > > > > > > > >>>>>>>>>>>>> maybe
> > > > > > > > >>>>>>>>>>>>>> we can resolve this by adding a qualification to
> > > > > > > > >> the
> > > > > > > > >>>> name
> > > > > > > > >>>>>>> (like
> > > > > > > > >>>>>>>>>>> Andrew
> > > > > > > > >>>>>>>>>>>>>> suggested, e.g., VECTOR_EMBEDDING or
> > > FINITE_VECTOR)
> > > > > > > > >>> to
> > > > > > > > >>>>>> avoid
> > > > > > > > >>>>>>>>>>> confusion
> > > > > > > > >>>>>>>>>>>>> with
> > > > > > > > >>>>>>>>>>>>>> broader ML use cases. Alternatively, we could
> > > > > > > > >>>> introduce a
> > > > > > > > >>>>>>>> boolean
> > > > > > > > >>>>>>>>>>> flag
> > > > > > > > >>>>>>>>>>>>>> (e.g., optional bool VectorType.finite) as
> > > proposed
> > > > > > > > >>>>>> earlier.
> > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>> On Fri, Sep 25, 2026 at 3:52 AM Andrew Lamb <
> > > > > > > > >>>>>>>>>> andrewlamb11@gmail.com>
> > > > > > > > >>>>>>>>>>>>>> wrote:
> > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>> I am a little confused about the points of
> > > > > > > > >>>> disagreement
> > > > > > > > >>>>>>> here.
> > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>> Is it that the name `VECTOR` is too general for
> > > > > > > > >> the
> > > > > > > > >>>>>>> specific
> > > > > > > > >>>>>>>>> use
> > > > > > > > >>>>>>>>>>> case
> > > > > > > > >>>>>>>>>>>>> of
> > > > > > > > >>>>>>>>>>>>>>> fixed length arrays with non-finite values used
> > > > > > > > >> for
> > > > > > > > >>>>>> vector
> > > > > > > > >>>>>>>>>>>> embeddings?
> > > > > > > > >>>>>>>>>>>>>> If
> > > > > > > > >>>>>>>>>>>>>>> so, maybe we could add a qualification to the
> > > > > > > > >> name
> > > > > > > > >>>>>>>>>> (FINITE_VECTOR?
> > > > > > > > >>>>>>>>>>>>>>> FEATURE_VECTOR? VECTOR_EMBEDDING?)
> > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>> It seems like there is precedent for defining
> > > > > > > > >>> logical
> > > > > > > > >>>>>> types
> > > > > > > > >>>>>>>>> that
> > > > > > > > >>>>>>>>>>>>> specify
> > > > > > > > >>>>>>>>>>>>>>> certain physical types and restrict their
> > > > > > > > >> contents
> > > > > > > > >>> to
> > > > > > > > >>>>>> help
> > > > > > > > >>>>>>>>>> readers
> > > > > > > > >>>>>>>>>>>>>>> interpret meaning, for example string[1] or
> > JSON;
> > > > > > > > >>>>>> Defining
> > > > > > > > >>>>>>>> one
> > > > > > > > >>>>>>>>>> for
> > > > > > > > >>>>>>>>>>>>>> vector
> > > > > > > > >>>>>>>>>>>>>>> embeddings seems reasonable enough to me.
> > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>> Andrew
> > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>> [1]:
> > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>
> > > > > > > > >>>>>>>>>>
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>>
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>
> > > > > > > > >>>>>
> > > > > > > > >>>>
> > > > > > > > >>>
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#string
> > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>> On Thu, Sep 24, 2026 at 3:34 PM Daniel Weeks <
> > > > > > > > >>>>>>>>> dweeks@apache.org>
> > > > > > > > >>>>>>>>>>>>> wrote:
> > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>> Well, Parquet is not a single-purpose format.
> > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>> Vector stores can choose whatever policy fits
> > > > > > > > >>>> their
> > > > > > > > >>>>>>>> usage,
> > > > > > > > >>>>>>>>>> but
> > > > > > > > >>>>>>>>>>>>> we're
> > > > > > > > >>>>>>>>>>>>>>>>> talking about a general-purpose file format
> > > > > > > > >>> that
> > > > > > > > >>>>> aims
> > > > > > > > >>>>>>> to
> > > > > > > > >>>>>>>> be
> > > > > > > > >>>>>>>>>>>> broadly
> > > > > > > > >>>>>>>>>>>>>>>>> applicable.
> > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>> I agree that the format is intended to be
> > > > > > > > >> general
> > > > > > > > >>>>>>> purpose,
> > > > > > > > >>>>>>>>> but
> > > > > > > > >>>>>>>>>>>> types
> > > > > > > > >>>>>>>>>>>>>>> within
> > > > > > > > >>>>>>>>>>>>>>>> the format define specific representation and
> > > > > > > > >>>>> semantics
> > > > > > > > >>>>>>> to
> > > > > > > > >>>>>>>>>>> achieve
> > > > > > > > >>>>>>>>>>>>> that
> > > > > > > > >>>>>>>>>>>>>>>> goal.
> > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>> "Vector" has a more general sense than ML and
> > > > > > > > >>>> it's
> > > > > > > > >>>>>>> quite
> > > > > > > > >>>>>>>>>>> trivial
> > > > > > > > >>>>>>>>>>>> to
> > > > > > > > >>>>>>>>>>>>>>>>> produce vectors with non-finite values using
> > > > > > > > >>> e.g.
> > > > > > > > >>>>>> NumPy
> > > > > > > > >>>>>>>> or
> > > > > > > > >>>>>>>>>>> Pandas
> > > > > > > > >>>>>>>>>>>>> (or
> > > > > > > > >>>>>>>>>>>>>>>>> any other piece of dataframe or scientific
> > > > > > > > >>>>> computing
> > > > > > > > >>>>>>>>>> software).
> > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>> The justification for restricting non-finite
> > > > > > > > >>>> values
> > > > > > > > >>>>> is
> > > > > > > > >>>>>>>>>> directly
> > > > > > > > >>>>>>>>>>>>>> related
> > > > > > > > >>>>>>>>>>>>>>> to
> > > > > > > > >>>>>>>>>>>>>>>> the semantic meaning of a vector.  Vectors
> > have
> > > > > > > > >>>>>> specific
> > > > > > > > >>>>>>>>>>>>>> characteristics
> > > > > > > > >>>>>>>>>>>>>>>> (dimension, magnitude, direction, etc).
> > > > > > > > >>> Non-finite
> > > > > > > > >>>>>>> values
> > > > > > > > >>>>>>>>> are
> > > > > > > > >>>>>>>>>>>>> invalid
> > > > > > > > >>>>>>>>>>>>>>> and
> > > > > > > > >>>>>>>>>>>>>>>> break those characteristics.  I'd liken it to
> > > > > > > > >>>> storing
> > > > > > > > >>>>>>>>> non-utf8
> > > > > > > > >>>>>>>>>>>> values
> > > > > > > > >>>>>>>>>>>>>> in
> > > > > > > > >>>>>>>>>>>>>>> a
> > > > > > > > >>>>>>>>>>>>>>>> string and still calling it a string.
> > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>> That also differentiates a vector from a fixed
> > > > > > > > >>>> length
> > > > > > > > >>>>>>> list,
> > > > > > > > >>>>>>>>>> which
> > > > > > > > >>>>>>>>>>>> is
> > > > > > > > >>>>>>>>>>>>>> more
> > > > > > > > >>>>>>>>>>>>>>>> general and does not carry the same semantic
> > > > > > > > >>>> meaning.
> > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>> I also don't think we need to bundle all
> > > > > > > > >>> similarly
> > > > > > > > >>>>>> shaped
> > > > > > > > >>>>>>>> use
> > > > > > > > >>>>>>>>>>> cases
> > > > > > > > >>>>>>>>>>>>>> into
> > > > > > > > >>>>>>>>>>>>>>>> the same solution (e.g. flxed size list).
> > > > > > > > >> Vector
> > > > > > > > >>>> is
> > > > > > > > >>>>> a
> > > > > > > > >>>>>>> well
> > > > > > > > >>>>>>>>>>>>> established
> > > > > > > > >>>>>>>>>>>>>>>> data type at this point and aligns with the
> > > > > > > > >>> general
> > > > > > > > >>>>>>> purpose
> > > > > > > > >>>>>>>>>>> nature
> > > > > > > > >>>>>>>>>>>> of
> > > > > > > > >>>>>>>>>>>>>> the
> > > > > > > > >>>>>>>>>>>>>>>> format (many other formats and databases
> > > > > > > > >> natively
> > > > > > > > >>>>>> support
> > > > > > > > >>>>>>>>>> this).
> > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>> -Dan
> > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>> On Thu, Sep 24, 2026 at 11:55 AM Antoine
> > > > > > > > >> Pitrou <
> > > > > > > > >>>>>>>>>>>> antoine@python.org>
> > > > > > > > >>>>>>>>>>>>>>>> wrote:
> > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>> Le 24/09/2026 à 20:11, Russell Spitzer a
> > > > > > > > >> écrit
> > > > > > > > >>> :
> > > > > > > > >>>>>>>>>>>>>>>>>> I think something I haven't seen yet is the
> > > > > > > > >>>> clear
> > > > > > > > >>>>>>>>> use-case
> > > > > > > > >>>>>>>>>>> for
> > > > > > > > >>>>>>>>>>>>>>>> non-finite
> > > > > > > > >>>>>>>>>>>>>>>>>> values within a vector type. There has
> > > > > > > > >> been a
> > > > > > > > >>>> lot
> > > > > > > > >>>>>> of
> > > > > > > > >>>>>>>>>>>> theoretical
> > > > > > > > >>>>>>>>>>>>>>>>> discussion
> > > > > > > > >>>>>>>>>>>>>>>>>> here about use cases but what toolings
> > > > > > > > >>>> currently
> > > > > > > > >>>>>>>> produce
> > > > > > > > >>>>>>>>>>> these
> > > > > > > > >>>>>>>>>>>>>>> vectors
> > > > > > > > >>>>>>>>>>>>>>>>> and
> > > > > > > > >>>>>>>>>>>>>>>>>> how do they function in downstream engines?
> > > > > > > > >>> I'd
> > > > > > > > >>>>>> like
> > > > > > > > >>>>>>> to
> > > > > > > > >>>>>>>>>> limit
> > > > > > > > >>>>>>>>>>>>>>>> introducing
> > > > > > > > >>>>>>>>>>>>>>>>>> types that have a wide range of values
> > > > > > > > >> which
> > > > > > > > >>>> are
> > > > > > > > >>>>>>> simply
> > > > > > > > >>>>>>>>> are
> > > > > > > > >>>>>>>>>>>>> failure
> > > > > > > > >>>>>>>>>>>>>>>> modes
> > > > > > > > >>>>>>>>>>>>>>>>>> of different flavors
> > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>> "Vector" has a more general sense than ML and
> > > > > > > > >>>> it's
> > > > > > > > >>>>>>> quite
> > > > > > > > >>>>>>>>>>> trivial
> > > > > > > > >>>>>>>>>>>> to
> > > > > > > > >>>>>>>>>>>>>>>>> produce vectors with non-finite values using
> > > > > > > > >>> e.g.
> > > > > > > > >>>>>> NumPy
> > > > > > > > >>>>>>>> or
> > > > > > > > >>>>>>>>>>> Pandas
> > > > > > > > >>>>>>>>>>>>> (or
> > > > > > > > >>>>>>>>>>>>>>>>> any other piece of dataframe or scientific
> > > > > > > > >>>>> computing
> > > > > > > > >>>>>>>>>> software).
> > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>> It's not like the general concept of a vector
> > > > > > > > >>> is
> > > > > > > > >>>> a
> > > > > > > > >>>>>> IETF
> > > > > > > > >>>>>>>>>>> standard
> > > > > > > > >>>>>>>>>>>> or
> > > > > > > > >>>>>>>>>>>>>>>>> anything, so analogies with JSON or UTF8
> > > > > > > > >> (where
> > > > > > > > >>>> the
> > > > > > > > >>>>>>>> values
> > > > > > > > >>>>>>>>>> are
> > > > > > > > >>>>>>>>>>>>>> expected
> > > > > > > > >>>>>>>>>>>>>>>>> to be valid according to the respective
> > > > > > > > >> specs)
> > > > > > > > >>> do
> > > > > > > > >>>>> not
> > > > > > > > >>>>>>>>> apply.
> > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>> This discussion would IMHO have a better
> > > > > > > > >> chance
> > > > > > > > >>>> of
> > > > > > > > >>>>>>>>> converging
> > > > > > > > >>>>>>>>>>> if
> > > > > > > > >>>>>>>>>>>> we
> > > > > > > > >>>>>>>>>>>>>>>>> separated the two orthogonal concerns of:
> > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>> 1. representing fixed-length list/vector data
> > > > > > > > >>> in
> > > > > > > > >>>>>>> Parquet
> > > > > > > > >>>>>>>>>> (with
> > > > > > > > >>>>>>>>>>> a
> > > > > > > > >>>>>>>>>>>>>>>>> potential limitation to non-nested types)
> > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>> 2. representing the information that a given
> > > > > > > > >>>> column
> > > > > > > > >>>>>> (or
> > > > > > > > >>>>>>>>>> column
> > > > > > > > >>>>>>>>>>>>> chunk,
> > > > > > > > >>>>>>>>>>>>>>>>> etc.) does not contain non-finite values
> > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>> ... both of which are useful on their own
> > > > > > > > >>> right,
> > > > > > > > >>>>> but
> > > > > > > > >>>>>> do
> > > > > > > > >>>>>>>> not
> > > > > > > > >>>>>>>>>>> have
> > > > > > > > >>>>>>>>>>>> to
> > > > > > > > >>>>>>>>>>>>>> be
> > > > > > > > >>>>>>>>>>>>>>>>> conflated and bundled together.
> > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>> Regards
> > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>> Antoine.
> > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>
> > > > > > > > >>>>>>>>>>
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>>
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>
> > > > > > > > >>>>>
> > > > > > > > >>>>
> > > > > > > > >>>
> > > > > > > > >>
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >

Re: [DISCUSS] Numeric Vector type: physical representation (outcome of focused call)

[Ryan Blue <bl...@apache.org>]

Jan,

I agree that we _could_ model vectors as an orthogonal finiteness property
for elements in a fixed-length list. I don't know how useful that would be.
I've not had anyone ask for that, and I'm not aware of databases with
finite floating point types. I think it makes more sense to move forward
with the existing FINITE_VECTOR that is proposed.

For the logical type argument, it sounds like the issue is that you want to
fall back to readers treating vectors as LIST. I don't see a good way to
make that happen, but it also isn't needed by the table format use case
because we are introducing this as a new type. Do you have a use case where
you are going to mix FINITE_VECTOR with LIST files and this will cause a
problem?

I don't think that we've had an expectation for logical types in the past
that they produce a particular in-memory representation. Maybe we could
allow setting both `LIST` and `FINITE_VECTOR` logical types at the same
time? This kind of compatibility hasn't been a priority as far as I'm
aware, but if it is important to you we can consider it.

Ryan

On Wed, Oct 21, 2026 at 1:13 PM Jan Finis <jp...@gmail.com> wrote:

>  assuming what will be proposed is a new logical type FINITE_VECTOR that
> annotates the same column that would have been annotated as LIST before:
>
> What I find somewhat suboptimal of FINITE_VECTOR is that finiteness is part
> of the vector type. Isn't the finiteness rather a property of the contained
> floats? Such a modeling would be more composable, as it would for example
> also allow to have normal (non-vector) float columns that are restricted to
> be finite. So, if we do think that we want finiteness in the type system, I
> would rather cut it into two parts: A fixed size vector concept, and a new
> way of labeling float columns finite. The vector type would then just boil
> down to be a fixed size list / we could just give the list type an optional
> fixed size field. Then, the column contained within it would be a float
> tagged as finite.
>
> It was mentioned that a logical type annotating a list of float would be
> forward compatible, but - depending on the modeling - that is not
> necessarily true: as logical types in Parquet do not support subtyping, a
> column cannot be a list of float and a (finite) vector at the same time,
> but only one of the two. It depends on how the type will be modeled in
> thrift in the end, but if it is a new union member of `LogicalType`, then
> legacy engines will not understand that such a vector can also just be
> interpreted as a list of floats. They will just see a nested column type of
> a logical type they don't understand, so they will likely reject reading
> this column (at least that's what the engine I'm maintaining would be
> doing).
>
> In contrast, if we just give the list type an optional fixed size and float
> an optional finiteness, then the resulting thrift is actually fully forward
> compatible: Legacy engines will just ignore the fixed length and treat the
> list as variable length and they will ignore the finiteness and treat the
> element type as float.
>
> (A side note, which most people here may already be aware of, but it's good
> to repeat it: The not-NULL property doesn't need new type support at all.
> The normal type system of list already allows setting the contained element
> type to be optional or required. If it is set to be required, then NULLs
> are not allowed.)
>
> Cheers,
> Jan
>
>
>
>
>
> Am Di., 20. Okt. 2026 um 20:37 Uhr schrieb Russell Spitzer <
> russell.spitzer@gmail.com>:
>
> > I support proceeding with the two types. I honestly think the cost of
> > adding new logical types is overestimated. If we have as many interested
> > parties as have appeared in this thread we already know there is a large
> > group of engineers ready to do the work and engines ready to consume (and
> > produce) the type. The fact that implementations currently use
> heuristics,
> > sampling (I loved Gunnar's talk on his hardwood work on this at CoC
> > Glasgow) or custom annotations to achieve the same performance benefits
> > makes me think we should just create the type and move on.
> >
> > On Mon, Oct 19, 2026 at 7:21 PM Ryan Blue <bl...@apache.org> wrote:
> >
> > > An annotation on list<float> is useful because:
> > > - We want to correctly annotate what is being stored
> > > - We want to unlock optimizations when each non-null list has the same
> > > number of elements
> > > - We want to have writer guarantees
> > >
> > > I don't think that logical type compatibility is a good argument. We
> want
> > > to avoid a case where we have to treat all binary like a string. And
> > > because we are introducing this in table formats after standardizing it
> > > here, we can do that reliably.
> > >
> > > As I said in my last email, I think there's a difference between
> > finiteness
> > > and general "invariants" you might want. Vectors aren't vectors if they
> > > don't have a length and a direction. Other invariants don't work the
> same
> > > way. A zero-length vector is still a valid vector and can be used, just
> > not
> > > for certain distance metrics.
> > >
> > > I also want to clarify that when we looked into this more closely and
> > > recognized the mismatch between the two separate use cases, I offered
> to
> > > split out the finite vector work so that we didn't cause any headaches
> > for
> > > people who are interested in fixed-length list. It sounds like not
> > > splitting out the use cases is causing a lot of headache. Vectors for
> our
> > > use case need a finiteness guarantee at write time, and fixed-length
> > lists
> > > need to store NaN and other values.
> > >
> > > It's okay to have two types and we should for two different use cases.
> > > Otherwise, we are forcing one to make an unnecessary compromise. I
> don't
> > > want to force everyone to exclude NaN and infinite values from
> > fixed-length
> > > lists. Seems reasonable that we should also be able to have vectors
> that
> > > exclude those values, since that is how they are used and we have to
> also
> > > guarantee it for exchanging data with other existing databases.
> > >
> > > Ryan
> > >
> > > On Mon, Oct 19, 2026 at 3:12 AM Will Edwards via dev <
> > > dev@parquet.apache.org>
> > > wrote:
> > >
> > > > It reads well. Three small fixes:
> > > >
> > > > 1. **"back in May or so"** is ambiguous. The
> > fixed-size-list-for-tensors
> > > > framing is from Rok's opening post in May 2024. The May 2026 design
> doc
> > > was
> > > > already mostly about performance and layout. "Originally" avoids
> anyone
> > > > correcting the date.
> > > > 2. **"Spark's sparse feature vectors routinely use NaN"** isn't quite
> > > > right. The NaN-for-missing behaviour comes from `VectorAssembler` and
> > > > `Imputer` and applies to Spark ML vectors generally, dense or sparse.
> > > > "Spark's feature vectors, dense and sparse," fixes it.
> > > > 3. There are typos: "Iirc" should be "IIRC", and "Spark, BigQuery and
> > > > DuckDB and pretty much…" has a doubled "and".
> > > >
> > > > The last sentence of the invariants paragraph could be clearer.
> > "Although
> > > > that doesn't mean readers shouldn't check on read" has a double
> > negative
> > > > and is easy to misread. "Though careful readers will still check on
> > read"
> > > > is simpler.
> > > >
> > > > With those applied:
> > > >
> > > > ---
> > > >
> > > > Hi all,
> > > >
> > > > The scope of this discussion keeps moving :). IIRC it originally was
> > > about
> > > > fixed-size lists for ML tensors, became a performance question, and
> is
> > > now
> > > > about whether vectors are mathematically valid. Each step made sense,
> > > but I
> > > > don't think we have consensus on where we've ended up?
> > > >
> > > > I think there is much useful work in what we left behind. Many ML
> > vectors
> > > > aren't embeddings for search. E.g. Spark's feature vectors, dense and
> > > > sparse, routinely use NaN to mark missing values, and quantised
> formats
> > > > pack codes with scales in ways Parquet can't describe today.
> > ML-oriented
> > > > formats like Lance are moving towards tensors, quantisation and
> > indexes.
> > > A
> > > > type that told engines how to interpret that kind of data would
> enable
> > > new
> > > > things. An annotation on LIST<FLOAT> doesn't.
> > > >
> > > > My big concern with the current focus on embeddings needing a logical
> > > type
> > > > is compatibility. Today LIST<FLOAT> is the common format for
> > embeddings,
> > > > and Spark, BigQuery, DuckDB and pretty much every other library I can
> > > think
> > > > of offer distance functions on plain lists, so any engine can use any
> > > file.
> > > > If engines start requiring a VECTOR annotation, existing files and
> > > anything
> > > > rewritten by a tool that doesn't know the type stop working. If they
> > > don't
> > > > require it, the annotation adds nothing. We've seen this before with
> > > > strings written without the UTF8 annotation, which engines still have
> > to
> > > > cope with years later.
> > > >
> > > > On invariants, I agree some users may want the writer to check what
> > they
> > > > write as they write it, but that can be a writer option rather than a
> > > type.
> > > > Readers that want evidence could get it from extra statistics or
> > > metadata,
> > > > though careful readers will still check on read.
> > > >
> > > > If a finite fixed-length vector contract is useful to table formats,
> > > > perhaps Iceberg is the place for it, with Parquet storing plain
> lists.
> > > I'd
> > > > raise the same worry there, though: if tables start rejecting
> untagged
> > > > columns, we've just moved the split up a layer.
> > > >
> > > > Best,
> > > > Will
> > > >
> > > > On Sat, 17 Oct 2026 at 00:10, Ryan Blue <bl...@apache.org> wrote:
> > > >
> > > > > I agree with much of what Dan said, but I'll add some to this
> thread
> > as
> > > > > well.
> > > > >
> > > > > First, I want to address the arguments that were pointed out:
> > > > > 1. Other vector databases have finiteness requirements
> > > > > 2. "Vectors are the hot stuff right now" -- let's jump on the
> > bandwagon
> > > > > 3. A general solution will take too long, finiteness will make
> faster
> > > > > progress
> > > > > 4. To give a guarantee to readers that no INF/NaN/NULL will be in
> the
> > > > > vector
> > > > >
> > > > > I don't think these represent our actual motivation or position,
> > which
> > > > > makes them function as a straw-man. This is probably not what was
> > > > > intended, but I want to make sure that the misunderstanding
> > > > > doesn't continue. For one thing, we aren't saying that we need
> finite
> > > > > vectors because they are "hot stuff".
> > > > >
> > > > > We want to build tools that use them on top of Parquet. I think it
> > > should
> > > > > also be clear that we aren't advocating for finite elements blindly
> > > just
> > > > > because other engines do (cargo cult design!). There are real
> > > > > interoperability issues if we choose to go a different direction so
> > > this
> > > > is
> > > > > a "Chesterton's Fence" situation. It's important to understand why
> > > > > those databases
> > > > > made that choice. I've also pointed this out as confirmation that
> > we're
> > > > > going down a reasonable and established path.
> > > > >
> > > > > The third argument sounds like a simple misunderstanding. We're
> > > pursuing
> > > > a
> > > > > logical type right now because that's forward compatible and we
> want
> > to
> > > > > label the data properly and set expectations for the type. Physical
> > > type
> > > > > changes break forward compatibility and need to be released in a
> new
> > > > format
> > > > > version . I think this is the source of the confusion since it has
> > been
> > > > in
> > > > > the same discussion, but the choice about finiteness is separate.
> > > > >
> > > > > That brings us to the last argument, whether there is value in
> > writers
> > > > > participating in guarantees. I think there's value in adding
> > > requirements
> > > > > for writers. It's up to readers how they want to use those
> > > requirements,
> > > > > but a writer requirement allows a policy of "garbage in garbage
> out"
> > > > rather
> > > > > than being forced to handle bad data. Why would a reader need to
> run
> > > > > validation on every element just in case a writer had a bug? And
> why
> > > > should
> > > > > the potential for writer bugs mean that it has to be the reader's
> > > > problem?
> > > > > If a writer produces bad data, we can absolutely reject it and tell
> > the
> > > > > writer to fix the bug.
> > > > >
> > > > > I'm also puzzled by the assertion that writer guarantees have
> value,
> > > but
> > > > > only if there are other, more specific guarantees in addition. This
> > > seems
> > > > > to cede the point that a contract between readers and writers makes
> > > > sense,
> > > > > but only if it is a _specific_ contract. But the proposed additions
> > > don't
> > > > > make sense in the same way.
> > > > >
> > > > > Take, for example, the suggestion to also disallow zero-length
> > vectors.
> > > > At
> > > > > a glance this seems similar, but I think it assumes a specific
> > context
> > > > and
> > > > > is better suited to be a constraint for some use cases rather than
> > part
> > > > of
> > > > > what is considered valid for the type. That's because zero-length
> > > vectors
> > > > > are valid vectors with a length. You can calculate Euclidean or
> > > Manhattan
> > > > > distance and use them in many contexts. They don't work if you're
> > > > > normalizing to a length of 1, but it's a choice to normalize and
> use
> > a
> > > > dot
> > > > > product to calculate cosine distance. And at this point, we're
> > talking
> > > > > about the usefulness of restrictions for the type, not a Platonic
> > > ideal.
> > > > >
> > > > > I think the restriction that makes the most sense is that vectors
> > have
> > > a
> > > > > length and can be compared in distance calculations. 0-length
> vectors
> > > are
> > > > > valid because they point somewhere and have a length. We shouldn't
> > > > restrict
> > > > > uses to only cosine distance over normalized vectors. Those are use
> > > case
> > > > > dependent choices and there are other valid ones. I also don't
> think
> > > that
> > > > > the model used to produce a vector is part of the vector type, just
> > > like
> > > > > the method used to produce a metric doesn't define its type: cosine
> > > > > similarity produces values between -1 and 1, but we store it in a
> > > > `float`,
> > > > > not a `float(cosine)`.
> > > > >
> > > > > Why does it make sense to have a finiteness restriction as part of
> > the
> > > > > type? Because that's what a (mathematical) vector is. A simple
> search
> > > > gives
> > > > > a pretty common definition: "A vector is an object that has both
> > > > magnitude
> > > > > and a direction". If a dimension/element is NaN, null, or infinite,
> > the
> > > > > vector doesn't point anywhere and it doesn't have a magnitude. Not
> > only
> > > > can
> > > > > it not be used, it doesn't meet the most basic definition of the
> > type.
> > > > >
> > > > > Ryan
> > > > >
> > > > > On Thu, Oct 15, 2026 at 1:07 AM Jan Finis <jp...@gmail.com>
> wrote:
> > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > Daniel,
> > > > > >
> > > > > > alright, so no physical optimizations intended.  We only want
> this
> > > for
> > > > > > semantic reasons, not for any significant physical optimization.
> If
> > > > we're
> > > > > > all on the same page here, that is good 👍.
> > > > > >
> > > > > > To phrase my remaining problem with this more concretely:
> > > > > > *Let's say we add this type. What cool new stuff can we now do
> that
> > > we
> > > > > > couldn't do before?*
> > > > > > That is the litmus test that a type has to pass to be useful
> IMHO.
> > > And
> > > > > here
> > > > > > I can't come up with a tangible use case that wasn't possible
> > before.
> > > > > > Especially if we one day have fixed size lists in Parquet, the
> > > > remaining
> > > > > > niche for this type becomes so small, that the added complexity
> > isn't
> > > > > worth
> > > > > > the benefit IMHO.
> > > > > >
> > > > > > As I guess most people on here are active maintainers of a
> Parquet
> > > > > > implementation: Concretely, what would you be using this type for
> > if
> > > we
> > > > > had
> > > > > > it? Would you allow math only on fields tagged with this type?
> What
> > > if
> > > > > that
> > > > > > math yields INF or NaN? Not allowed?
> > > > > >
> > > > > > Especially if we also discuss having both VECTOR and
> FINITE_VECTOR.
> > > How
> > > > > > will you guys wire this up with the type systems of your engines?
> > Do
> > > > your
> > > > > > engines also have two types here? Do you type-decay a
> FINITE_VECTOR
> > > to
> > > > a
> > > > > > VECTOR once someone does a multiplication that could yield INF?
> By
> > > > > itself,
> > > > > > a FINITE_VECTOR sounds good. Holistically looked at from a type
> > > system
> > > > > > perspective, it comes with a lot of complexity. Do the benefits
> > > really
> > > > > > outweigh the complexity here? That's why I would rather have the
> > bare
> > > > > > minimum: A fixed size array (list) of floats or at least just a
> > > vector
> > > > > > without finite. Then all these detailed questions go away.
> > > > > >
> > > > > > I'll take this one step further... Why is there any debate about
> > this
> > > > at
> > > > > > > all? Why can't a type simply be added to the *specification* as
> > an
> > > > > > > administrative detail and be done?
> > > > > >
> > > > > >
> > > > > > Andrew,
> > > > > >
> > > > > > Parquet is supposed to be a lingua franca of data. The
> > specification
> > > is
> > > > > the
> > > > > > authoritative truth for this language and all implementations are
> > > > > expected
> > > > > > to implement as much as possible from this spec, so anything in
> the
> > > > spec
> > > > > > does causes implementation effort. So just adding anything to
> this
> > > spec
> > > > > > without debate is not a good idea. It is even the other way
> around:
> > > We
> > > > > need
> > > > > > to have way higher rigor with the spec than we need to have with
> > any
> > > > > > implementations. Implementations can be changed, the spec needs
> to
> > be
> > > > > > stable.
> > > > > >
> > > > > > A tag-only type that doesn't come with any restrictions can be
> > added
> > > > with
> > > > > > little effort. But here we want to tie restrictions to it (no
> > > NaN/Inf),
> > > > > so
> > > > > > we would expect implementations to enforce these. So there is
> some
> > > > > > implementation effort expected here.
> > > > > >
> > > > > > Also, even if we only wanted a tag type, I would still argue that
> > you
> > > > > > shouldn't add those without a debate. What if A wants XML, B
> wants
> > > > YAML,
> > > > > C
> > > > > > wants quaternions, D wants address, E wants phone number. If we
> > just
> > > > > added
> > > > > > any types that are "just tags", Parquet logical types would just
> > > > become a
> > > > > > trash dump of any type that anyone ever thought to be useful. So
> > > > rigorous
> > > > > > assessment of each new type candidate is necessary.
> > > > > >
> > > > > > Finally, many engines couple parquet with another type system
> > (e.g.,
> > > > > Arrow
> > > > > > or SQL). So even with a tag only type, you at least still have
> the
> > > > > > implementation burden of wiring this up with your engine's type
> > > system.
> > > > > >
> > > > > > So, just because logical types are "just" logical, that doesn't
> > mean
> > > we
> > > > > can
> > > > > > add them without any rigor.
> > > > > >
> > > > > >
> > > > > > A few naive questions:
> > > > > > > 1. Would a physical definition with NANs & Inf+/- allow
> efficient
> > > > > > > encodings like ALP?
> > > > > > > 2. If the physical representation allows values that make
> > encodings
> > > > > like
> > > > > > > ALP impossible, can a tighter Logical type allow different
> > > encodings?
> > > > > > > 3. Can we RLE or binpack a vector?
> > > > > >
> > > > > >
> > > > > > The simple general answer here is that all these things do not
> > > require
> > > > a
> > > > > > logical type. Writers can already do any such optimization if the
> > > > actual
> > > > > > data contains no NaNs & Infs. All these physical optimizations
> just
> > > > > require
> > > > > > no actual NaNs/Infs to be present in a page, they don't need a
> > > > guarantee
> > > > > > that the whole file won't contain any.
> > > > > >
> > > > > > Cheers,
> > > > > > Jan
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > Am Do., 15. Okt. 2026 um 04:28 Uhr schrieb <
> > kurtiscwright@gmail.com
> > > >:
> > > > > >
> > > > > > > Hi Everyone,
> > > > > > >
> > > > > > > Glad to see a Vector type being added. I have a few use cases I
> > > want
> > > > to
> > > > > > > pursue once we have an agreed upon definition.
> > > > > > >
> > > > > > > In regards to that definition discussion. I feel comfortable
> with
> > > the
> > > > > > idea
> > > > > > > of the semantic meaning Dan raises though I can’t shake a
> slight
> > > > > > discomfort
> > > > > > > (maybe FOMO) with limiting values in the physical definition.
> > > > > > >
> > > > > > > A few naive questions:
> > > > > > > 1. Would a physical definition with NANs & Inf+/- allow
> efficient
> > > > > > > encodings like ALP?
> > > > > > > 2. If the physical representation allows values that make
> > encodings
> > > > > like
> > > > > > > ALP impossible, can a tighter Logical type allow different
> > > encodings?
> > > > > > > 3. Can we RLE or binpack a vector?
> > > > > > >
> > > > > > > Best Regards,
> > > > > > > Kurtis C. Wright
> > > > > > > > On Oct 14, 2026, at 12:45, Daniel Weeks <dw...@apache.org>
> > > wrote:
> > > > > > > >
> > > > > > > > Hey Jan,
> > > > > > > >
> > > > > > > > There's a lot that I disagree with respect to the framing of
> > > these
> > > > > > > > arguments and how we should be thinking about introducing
> > vector
> > > > as a
> > > > > > > > logical type.  There's a repeated tendency to let the
> physical
> > > > > > > > representation or performance drive discussion around types
> as
> > > > > opposed
> > > > > > to
> > > > > > > > what it represents.
> > > > > > > >
> > > > > > > > In parquet, establishing semantic meaning is the purpose of
> > > logical
> > > > > > > types.
> > > > > > > > The logical type separates how to interpret a value from its
> > > > physical
> > > > > > > > representation.  We can represent a timestamp with INT64 or
> > even
> > > a
> > > > > > string
> > > > > > > > value, so why have a type for it? Even though a
> > FIXED_SIZED_LIST
> > > > and
> > > > > > > VECTOR
> > > > > > > > may have the exact same representation and performance, they
> > have
> > > > > > > different
> > > > > > > > semantic meaning.  Databases and engines across the ecosystem
> > > > already
> > > > > > > > recognize this differentiation and have separate
> > representations.
> > > > > The
> > > > > > > > choice to separate these isn't because the discussion is too
> > > > tedious;
> > > > > > > > they're fundamentally different things.
> > > > > > > >
> > > > > > > > The finite requirement is what makes a vector a reflection of
> > > what
> > > > it
> > > > > > > > represents.  As we've discussed in the thread, non-finite
> > values
> > > in
> > > > > the
> > > > > > > > representation violate the properties of a vector and the
> value
> > > > > ceases
> > > > > > to
> > > > > > > > be a vector.  On the other hand, a 0 vector is still a vector
> > > (you
> > > > > can
> > > > > > > > argue the utility of it, but it is a valid vector).
> > > > Implementations
> > > > > > can
> > > > > > > > choose what they validate or how to handle execution errors;
> > > that's
> > > > > an
> > > > > > > > implementation concern, not a requirement.  Operations
> > performed
> > > > on a
> > > > > > > > vector depend on the context in which they are used, but the
> > > > > > > representation
> > > > > > > > doesn't need to solve scenarios where that application
> doesn't
> > > make
> > > > > > > sense.
> > > > > > > >
> > > > > > > > The guarantees or context needed to use a vector do not need
> to
> > > be
> > > > > > > inherent
> > > > > > > > in the type for it to be a useful type.  We don't need the
> > > currency
> > > > > > used
> > > > > > > > for decimal or whether a floating point is a normalized value
> > as
> > > > part
> > > > > > of
> > > > > > > > the type to make use of those types.  This conflates the use
> > > cases
> > > > > with
> > > > > > > the
> > > > > > > > type system and unnecessarily attempts to draw the context
> into
> > > the
> > > > > > type
> > > > > > > > definition.  Similarly, you don't need to capture
> > normalization,
> > > > > model,
> > > > > > > > etc. as part of the type for it to be useful.
> > > > > > > >
> > > > > > > > So the reason we want to add a vector type (or FINITE_VECTOR)
> > is
> > > > > > because
> > > > > > > we
> > > > > > > > want to store vectors with the semantic meaning and
> > requirements
> > > > > based
> > > > > > on
> > > > > > > > what they represent.
> > > > > > > >
> > > > > > > > -Dan
> > > > > > > >
> > > > > > > >
> > > > > > > >> On Wed, Oct 14, 2026 at 1:22 AM Jan Finis <
> jpfinis@gmail.com>
> > > > > wrote:
> > > > > > > >>
> > > > > > > >> A bit late to the party, but here we go:
> > > > > > > >>
> > > > > > > >> I have the strong sentiment that the FINITE_VECTOR type as
> it
> > is
> > > > > > > currently
> > > > > > > >> being discussed will not be very useful and just increase
> > > > > > implementation
> > > > > > > >> complexity. The arguments for the type are all quite weak
> > > (details
> > > > > see
> > > > > > > >> below). The FINITE part doesn't seem to yield any value and
> is
> > > > not a
> > > > > > > useful
> > > > > > > >> restriction by itself. The vector (i.e. fixed length) part
> > > should
> > > > > > > rather be
> > > > > > > >> generalized instead of making a one-off thing. In general,
> the
> > > > > > existing
> > > > > > > >> types are already working well for uncompressed vectors. The
> > > real
> > > > > > > challenge
> > > > > > > >> that would bring real benefits is supporting quantization.
> > This
> > > is
> > > > > > where
> > > > > > > >> you get orders of magnitude improvements to storage space
> and
> > > > > > processing
> > > > > > > >> speed.
> > > > > > > >>
> > > > > > > >> Let's first discuss the FINITE part. So far, the only
> > arguments
> > > I
> > > > > have
> > > > > > > read
> > > > > > > >> for this are:
> > > > > > > >>
> > > > > > > >> 1. many other vector databases have it
> > > > > > > >> 2. vectors are the hot stuff right now, so Parquet better
> > boast
> > > > > having
> > > > > > > >> "native" type support for vectors
> > > > > > > >> 3. by focussing on this common special case, we can make
> rapid
> > > > > > progress,
> > > > > > > >> while a general solution would take too long to specify
> > > > > > > >> 4. to give a guarantee to readers that no INF/NaN/NULL will
> be
> > > in
> > > > > the
> > > > > > > >> vector
> > > > > > > >>
> > > > > > > >> the first two arguments are quite weak by themselves. Vector
> > > > > databases
> > > > > > > are
> > > > > > > >> very specialized engines. Such engines can of course be very
> > > > > > > opinionated on
> > > > > > > >> their data types. A general format like Parquet must instead
> > > weigh
> > > > > > > whether
> > > > > > > >> a special type is worth the added complexity over a more
> > > general,
> > > > > > widely
> > > > > > > >> applicable type. In this case, it is IMHO not at all. And
> just
> > > > > having
> > > > > > a
> > > > > > > >> type called "vector" for advertisement reasons to make
> Parquet
> > > > > appear
> > > > > > > more
> > > > > > > >> modern seems a non-goal in the first place.
> > > > > > > >>
> > > > > > > >> The third "time to market" argument is valid for systems
> that
> > > > don't
> > > > > > > need a
> > > > > > > >> stable interface. It is a very dangerous attitude for
> formats
> > > like
> > > > > > > Parquet
> > > > > > > >> that must be stable long term and should have a
> consolidated,
> > > > > concise,
> > > > > > > >> non-redundant type system. For such long term storage
> formats,
> > > we
> > > > > need
> > > > > > > to
> > > > > > > >> take the time to build the right thing, as any shortcuts we
> > take
> > > > > will
> > > > > > > stay
> > > > > > > >> as blemishes in the format forever. If I read the discussion
> > > > > > correctly,
> > > > > > > >> this is actually the main argument as of now: We have found
> > that
> > > > it
> > > > > is
> > > > > > > hard
> > > > > > > >> to align on how a fixed size list type should look like, so
> we
> > > > want
> > > > > to
> > > > > > > go
> > > > > > > >> the easier route of a specialized type. This seems like a
> > > > > > > >> dangerous shortcut fueled by feature recency bias. Ask
> > yourself:
> > > > > Would
> > > > > > > we
> > > > > > > >> want this, if Parquet already had a fixed size list type?
> > > > > > > >>
> > > > > > > >> So finally, the only technical argument is the fourth one:
> "It
> > > > > signals
> > > > > > > >> semantics and ensures that no bad data lands in the
> format". I
> > > see
> > > > > two
> > > > > > > big
> > > > > > > >> problems with this:
> > > > > > > >>
> > > > > > > >> 1. Readers *still* need to test for NaN / INF because
> Parquet
> > > > files
> > > > > > may
> > > > > > > be
> > > > > > > >> corrupt. Nothing can stop a malicious or buggy writer from
> > > > > physically
> > > > > > > >> writing NaN or INF. *So you save no single line of code in a
> > > > reader.
> > > > > > You
> > > > > > > >> save no branches, as you still need to handle the INF/NaN
> > case.*
> > > > So
> > > > > > this
> > > > > > > >> doesn't help with performance at all. The only difference
> is,
> > > that
> > > > > the
> > > > > > > >> error message changes from "found an invalid INF value in
> the
> > > > data"
> > > > > to
> > > > > > > >> "Your parquet file seems to be corrupt". It is actually
> quite
> > > the
> > > > > > > opposite,
> > > > > > > >> readers and writers get more complex having to handle this
> new
> > > > type.
> > > > > > > >>
> > > > > > > >> If you now argue that it's not about performance, it's about
> > > > defined
> > > > > > > >> semantics and erroring out early in writers, then the second
> > > > problem
> > > > > > > >> arises:
> > > > > > > >>
> > > > > > > >> 2. The proposed no-NaN/INF/NULL guarantee is just one of
> > > multiple
> > > > > > > >> guarantees that you need for sensible vector processing. If
> > "no
> > > > > > > >> INF/NaN/NULL" was the only guarantee you needed, I would
> agree
> > > > that
> > > > > > > >> FINITE_VECTOR is useful, but it is by no means the only
> > > guarantee
> > > > > you
> > > > > > > need,
> > > > > > > >> instead, depending on the distance metric, you also need the
> > > > > following
> > > > > > > >> guarantees. E.g.:
> > > > > > > >>
> > > > > > > >>   - No vector may be the 0 vector, otherwise, e.g., cosine
> > > > > similarity
> > > > > > is
> > > > > > > >>   undefined. Cosine similarity is often used as default, so
> > this
> > > > is
> > > > > > > >> actually
> > > > > > > >>   quite important in practice.
> > > > > > > >>   - If we want to use a dot product, then all vectors need
> to
> > > > > > guarantee
> > > > > > > a
> > > > > > > >>   length of 1. This would actually be quite useful for
> > > performance
> > > > > > > >> reasons.
> > > > > > > >>   - The vectors must come from the same model as the vector
> > used
> > > > for
> > > > > > the
> > > > > > > >>   search. Otherwise, the search yields a numeric result, but
> > > that
> > > > > > result
> > > > > > > >> has
> > > > > > > >>   no sensible meaning.
> > > > > > > >>
> > > > > > > >> So, what use is a type that just guarantees part of the
> things
> > > you
> > > > > > need
> > > > > > > for
> > > > > > > >> vector search but not others? You still need additional
> > > guarantees
> > > > > in
> > > > > > > >> addition to that type. What makes INF/NaN so important in
> > > > comparison
> > > > > > to,
> > > > > > > >> say, "no 0 vector present", that it warrants an own type,
> > while
> > > > the
> > > > > > > other
> > > > > > > >> guarantees do not? If we really want this type as a basis
> for
> > > > > > > cross-engine
> > > > > > > >> sharing of vectors, then it needs to give *all* guarantees.
> > > > > > > >>
> > > > > > > >> I'm not against a specialized vector type if it really
> carries
> > > > *all*
> > > > > > > >> information to be useful by itself. If it carried all that
> > > > > > information,
> > > > > > > >> then a reading engine could indeed validate that only
> > meaningful
> > > > > > vector
> > > > > > > >> searches are executed. But as long as it is just a
> half-baked
> > > > thing,
> > > > > > it
> > > > > > > is
> > > > > > > >> not useful by itself and just a burden on each implementer
> and
> > > > > > > maintainer.
> > > > > > > >> If we need extra information anyway, why not just stick to a
> > > > general
> > > > > > > type
> > > > > > > >> like VECTOR (without FINITE).
> > > > > > > >>
> > > > > > > >> If you're a proponent of FINITE_VECTOR, ask yourself (or
> tell
> > > us)
> > > > > what
> > > > > > > >> advantage you will *really* get over just a (potentially
> fixed
> > > > size)
> > > > > > > List
> > > > > > > >> of Float? As discussed, it's not performance. It's also not
> a
> > > > > semantic
> > > > > > > >> marker that is sufficient by itself. It's just a small hint
> > that
> > > > > > allows
> > > > > > > >> writers to error out once they see a NaN or INF in the
> input,
> > > > while
> > > > > > they
> > > > > > > >> will still happily write a bogus 0 vector. This seems to be
> of
> > > > zero
> > > > > > > >> benefit.
> > > > > > > >>
> > > > > > > >> Anecdotally, we currently handle petabytes of vectors in
> > > > production
> > > > > in
> > > > > > > >> Data360. We simply store them in a List of Float, with an
> > > > optimized
> > > > > > > R-Level
> > > > > > > >> reader implementation that rapidly checks that all vectors
> are
> > > the
> > > > > > same
> > > > > > > >> length. This solution works just fine. The overhead of
> R-level
> > > > > > decoding
> > > > > > > is
> > > > > > > >> barely measurable. The check is super fast and R-levels are
> > tiny
> > > > in
> > > > > > > >> comparison to the payload vectors, so also the added I/O is
> > > > > > negligible.
> > > > > > > >> This is also why I'm so firm in my sentiment: We are
> running a
> > > > > > > production
> > > > > > > >> grade system processing trillions of vectors right now and
> see
> > > no
> > > > > big
> > > > > > > >> issues with the types that Parquet currently provides.
> > > > > > > >>
> > > > > > > >> As I don't just want to be a naysayer and point out problems
> > > with
> > > > > the
> > > > > > > >> current design, here are the directions that I would propose
> > > > indeed:
> > > > > > > >>
> > > > > > > >>   - a general fixed size list type. Don't do shortcuts just
> > > > because
> > > > > > the
> > > > > > > >>   discussion is too tedious.
> > > > > > > >>   - a vector type that indeed carries enough information to
> be
> > > > > really
> > > > > > > >>   useful for vector search by itself. (e.g., model name /
> > > version,
> > > > > > > >> optional
> > > > > > > >>   guarantees for no 0 vector / normalized vector length).
> With
> > > > such
> > > > > a
> > > > > > > >> type,
> > > > > > > >>   you could really do well defined vector searches even if
> the
> > > > > writing
> > > > > > > >> engine
> > > > > > > >>   is completely different to the reading one.
> > > > > > > >>   - Standardize quantizations. This will be a real game
> > changer
> > > > for
> > > > > > > >>   performance.
> > > > > > > >>
> > > > > > > >> Cheers,
> > > > > > > >> Jan
> > > > > > > >>
> > > > > > > >>
> > > > > > > >>
> > > > > > > >>
> > > > > > > >>
> > > > > > > >>
> > > > > > > >>> Am Mi., 14. Okt. 2026 um 03:36 Uhr schrieb Gang Wu <
> > > > > ustcwg@gmail.com
> > > > > > >:
> > > > > > > >>>
> > > > > > > >>> I think separating them into FINITE_VECTOR and
> > > FIXED_LENGTH_LIST
> > > > > is a
> > > > > > > >> good
> > > > > > > >>> idea, though I think renaming FINITE_VECTOR to VECTOR looks
> > > more
> > > > > > > natural.
> > > > > > > >>> Should we move forward with proposing FINITE_VECTOR/VECTOR
> > > first?
> > > > > > > >>>
> > > > > > > >>> On Tue, Oct 13, 2026 at 6:26 PM Alkis Evlogimenos via dev <
> > > > > > > >>> dev@parquet.apache.org> wrote:
> > > > > > > >>>
> > > > > > > >>>> I suggest we go with VECTOR for the finite vector to keep
> > the
> > > > name
> > > > > > the
> > > > > > > >>> same
> > > > > > > >>>> as other engines/type systems. If we want a non-finite one
> > in
> > > > the
> > > > > > > >> future
> > > > > > > >>> we
> > > > > > > >>>> can name that something else or we add fixed size list
> that
> > > will
> > > > > > serve
> > > > > > > >>> the
> > > > > > > >>>> same purpose without a special name.
> > > > > > > >>>>
> > > > > > > >>>>
> > > > > > > >>>> On Thu, Oct 8, 2026 at 10:20 PM Ryan Blue <
> blue@apache.org>
> > > > > wrote:
> > > > > > > >>>>
> > > > > > > >>>>> A logical type would:
> > > > > > > >>>>> - Describe data for users and readers to understand what
> > the
> > > > file
> > > > > > > >>>> contains
> > > > > > > >>>>> - Signal that repetition and definition levels will
> > reflect a
> > > > > fixed
> > > > > > > >>>> number
> > > > > > > >>>>> of elements, which implementations can use to optimize
> > > decoding
> > > > > > > >>>>> - Guarantee that there are no null elements (because the
> > type
> > > > > > cannot
> > > > > > > >> be
> > > > > > > >>>>> applied to a list with optional elements)
> > > > > > > >>>>> - [FINITE_VECTOR] Guarantee that no elements are NaN or
> > > > Infinity
> > > > > > > >> values
> > > > > > > >>>> so
> > > > > > > >>>>> that the vectors have a finite length and are suitable
> for
> > > > > distance
> > > > > > > >>>>> calculations
> > > > > > > >>>>>
> > > > > > > >>>>> The finite value restriction is a restriction on writers
> to
> > > > > prevent
> > > > > > > >> bad
> > > > > > > >>>>> data from being written.
> > > > > > > >>>>>
> > > > > > > >>>>> Ignoring logical types that are not understood is a rule
> > for
> > > > > > > >> readers. A
> > > > > > > >>>>> reader can handle vectors and pass it on to another
> layer.
> > > For
> > > > > > > >>> instance,
> > > > > > > >>>> a
> > > > > > > >>>>> Parquet reader that doesn't support geometry passes
> values
> > as
> > > > > > > >> `binary`.
> > > > > > > >>>>> That binary value can be passed to a geometry UDF, even
> if
> > > the
> > > > > > > >>> underlying
> > > > > > > >>>>> engine and Parquet reader doesn't have support.
> > > > > > > >>>>>
> > > > > > > >>>>> A tool that doesn't understand FINITE_VECTOR can
> similarly
> > > > pass a
> > > > > > > >> list
> > > > > > > >>> of
> > > > > > > >>>>> floats along to a UDF. But it is still important to the
> > user
> > > > that
> > > > > > bad
> > > > > > > >>>>> values are not present in the data. In your example, bad
> > > values
> > > > > are
> > > > > > > >>>> vectors
> > > > > > > >>>>> that cause the centroid calculation to fail, like vectors
> > > with
> > > > > > > >> infinite
> > > > > > > >>>>> length.
> > > > > > > >>>>>
> > > > > > > >>>>> Ryan
> > > > > > > >>>>>
> > > > > > > >>>>> On Thu, Oct 8, 2026 at 12:54 PM Will Edwards via dev <
> > > > > > > >>>>> dev@parquet.apache.org>
> > > > > > > >>>>> wrote:
> > > > > > > >>>>>
> > > > > > > >>>>>> Could we clarify the use cases for a VECTOR annotation
> on
> > > > > > > >>> LIST<FLOAT>?
> > > > > > > >>>>>> Specifically, what does it make possible that's
> impossible
> > > > > today,
> > > > > > > >> and
> > > > > > > >>>>> what
> > > > > > > >>>>>> does it make easier that's currently hard?
> > > > > > > >>>>>>
> > > > > > > >>>>>> I'm struggling to find a strong answer. I'd like every
> > > engine
> > > > to
> > > > > > > >>> offer
> > > > > > > >>>>>> vector functions on lists of floats without needing a
> new
> > > > > logical
> > > > > > > >>> type.
> > > > > > > >>>>>>
> > > > > > > >>>>>> To be clear, I'm still enthusiastic about a vector type
> > that
> > > > > > > >>> describes
> > > > > > > >>>>> how
> > > > > > > >>>>>> to unpack and scale composite representations (quantised
> > > codes
> > > > > > plus
> > > > > > > >>>>> scales,
> > > > > > > >>>>>> codebooks, etc.), in the direction ML is going. That
> adds
> > > > > > > >> information
> > > > > > > >>>>>> engines don't have. A label on LIST<FLOAT> doesn't:
> > engines
> > > > > > already
> > > > > > > >>>>>> understand the physical type, and many already offer
> > vector
> > > > > > > >> functions
> > > > > > > >>>> on
> > > > > > > >>>>>> it. I don't see a performance benefit, and NaN/null/Inf
> > > checks
> > > > > are
> > > > > > > >>>> cheap
> > > > > > > >>>>>> enough that engines should do them anyway. The label
> also
> > > > > doesn't
> > > > > > > >> say
> > > > > > > >>>>> what
> > > > > > > >>>>>> you can do with the column. Knowing that a column holds
> > > > > embeddings
> > > > > > > >>>>> doesn't
> > > > > > > >>>>>> say how they can be used, just as zipcode and age are
> both
> > > > > > integers
> > > > > > > >>> but
> > > > > > > >>>>>> multiplying them is meaningless.
> > > > > > > >>>>>>
> > > > > > > >>>>>> My concern is the opposite effect. We're establishing
> that
> > > > > engines
> > > > > > > >>>> should
> > > > > > > >>>>>> ignore logical types they don't understand. So a tool
> that
> > > > > doesn't
> > > > > > > >>> know
> > > > > > > >>>>>> VECTOR will read the column as LIST<FLOAT>, let the user
> > > > compute
> > > > > > > >>>>> centroids
> > > > > > > >>>>>> (say), and write plain LIST<FLOAT> back out. If other
> > > engines
> > > > > > > >>> restrict
> > > > > > > >>>>>> embedding functions to VECTOR-annotated columns, that
> file
> > > > stops
> > > > > > > >>>> working
> > > > > > > >>>>>> with them, through no fault of the user. An annotation
> > that
> > > > > > doesn't
> > > > > > > >>>>> change
> > > > > > > >>>>>> what the data is shouldn't be able to make it unusable.
> > > > > > > >>>>>>
> > > > > > > >>>>>> Am I missing a use case?
> > > > > > > >>>>>>
> > > > > > > >>>>>> Best, Will
> > > > > > > >>>>>>
> > > > > > > >>>>>> On Thu, 8 Oct 2026 at 19:53, Steven Wu <
> > > stevenz3wu@gmail.com>
> > > > > > > >> wrote:
> > > > > > > >>>>>>
> > > > > > > >>>>>>> I also think a FINITE_VECTOR type brings a lot of
> values
> > to
> > > > > many
> > > > > > > >>> use
> > > > > > > >>>>>> cases.
> > > > > > > >>>>>>> It is would be great to make progress on the
> > FINITE_VECTOR,
> > > > and
> > > > > > > >> not
> > > > > > > >>>> to
> > > > > > > >>>>>> fit
> > > > > > > >>>>>>> every use cases into the FIXED_LENGTH_LIST.
> > > > > > > >>>>>>>
> > > > > > > >>>>>>> On Mon, Oct 5, 2026 at 1:26 PM Ryan Blue <
> > blue@apache.org>
> > > > > > > >> wrote:
> > > > > > > >>>>>>>
> > > > > > > >>>>>>>> Going through this thread again, it looks like there
> > are a
> > > > few
> > > > > > > >>>> things
> > > > > > > >>>>>>> that
> > > > > > > >>>>>>>> people generally agree on:
> > > > > > > >>>>>>>>
> > > > > > > >>>>>>>> - In the short term, we want to introduce a logical
> type
> > > or
> > > > > > > >>> types;
> > > > > > > >>>> a
> > > > > > > >>>>>>> better
> > > > > > > >>>>>>>> physical representation can be done later. We can
> focus
> > on
> > > > > > > >>> logical
> > > > > > > >>>>>> types.
> > > > > > > >>>>>>>> - There is precedent in logical types for restricting
> > > > values,
> > > > > > > >>> like
> > > > > > > >>>>> UTF8
> > > > > > > >>>>>>>> strings to contain only UTF-8 codepoints. I think we
> > also
> > > > > agree
> > > > > > > >>>> that
> > > > > > > >>>>> we
> > > > > > > >>>>>>>> could restrict vector elements to non-nested or
> > primitive
> > > > > > > >> types.
> > > > > > > >>>>>>>> - There are many examples of databases that restrict
> > their
> > > > > > > >> vector
> > > > > > > >>>>> types
> > > > > > > >>>>>>> to
> > > > > > > >>>>>>>> non-null, finite elements.
> > > > > > > >>>>>>>> - Some people want to have write-time guarantees that
> > > > vectors
> > > > > > > >>> have
> > > > > > > >>>>> only
> > > > > > > >>>>>>>> non-null, finite elements
> > > > > > > >>>>>>>> - Some people want to extend this work to include
> > tensors,
> > > > > > > >>> shorter
> > > > > > > >>>>>>> floating
> > > > > > > >>>>>>>> point representations, variable lengths, NaN or
> Infinity
> > > > > > > >>> elements,
> > > > > > > >>>>> etc.
> > > > > > > >>>>>>>>
> > > > > > > >>>>>>>> It sounds like we are nearly there, but have two
> halves
> > of
> > > > the
> > > > > > > >>>>>> community
> > > > > > > >>>>>>>> that care about different things. That just means we
> > > should
> > > > > > > >> have
> > > > > > > >>>>>> multiple
> > > > > > > >>>>>>>> types. It's entirely reasonable to want a type that
> > allows
> > > > NaN
> > > > > > > >>>>> values,
> > > > > > > >>>>>>> and
> > > > > > > >>>>>>>> it's also reasonable for those of us building
> databases,
> > > > table
> > > > > > > >>>>>> standards,
> > > > > > > >>>>>>>> and vector indexes to care about guarantees that
> > elements
> > > > are
> > > > > > > >>>> finite.
> > > > > > > >>>>>>>> There's more than enough examples to show that this
> is a
> > > > > > > >>> legitimate
> > > > > > > >>>>> use
> > > > > > > >>>>>>>> case.
> > > > > > > >>>>>>>>
> > > > > > > >>>>>>>> Since I don't see the concerns of the different use
> > cases
> > > > > > > >>> changing,
> > > > > > > >>>>> it
> > > > > > > >>>>>>>> looks like we need to find a compromise to move
> > forward. I
> > > > > > > >>> propose
> > > > > > > >>>>> that
> > > > > > > >>>>>>> we
> > > > > > > >>>>>>>> introduce a FINITE_VECTOR type and separately a
> > > > > > > >> FIXED_LENGTH_LIST
> > > > > > > >>>> or
> > > > > > > >>>>>>>> another VECTOR type that allows infinite and NaN
> values.
> > > > > > > >>>>>>>>
> > > > > > > >>>>>>>> Can we agree on a compromise? Is there a strong reason
> > to
> > > > try
> > > > > > > >> to
> > > > > > > >>>> fit
> > > > > > > >>>>>>> these
> > > > > > > >>>>>>>> use cases into a single type? What if that means not
> > > making
> > > > > > > >>>> progress?
> > > > > > > >>>>>>>>
> > > > > > > >>>>>>>> Ryan
> > > > > > > >>>>>>>>
> > > > > > > >>>>>>>> On Mon, Oct 5, 2026 at 12:15 PM Philipp Fischbeck via
> > dev
> > > <
> > > > > > > >>>>>>>> dev@parquet.apache.org> wrote:
> > > > > > > >>>>>>>>
> > > > > > > >>>>>>>>> Thinking about this more, I am also fine with pulling
> > out
> > > > the
> > > > > > > >>>>>> “NaN/Inf”
> > > > > > > >>>>>>>>> requirement. Either into a type boolean (my
> > preference),
> > > or
> > > > > > > >>>> putting
> > > > > > > >>>>>> no
> > > > > > > >>>>>>>>> restrictions on the Parquet level, allowing all IEEE
> > 754
> > > > > > > >> float
> > > > > > > >>>>>> numbers.
> > > > > > > >>>>>>>> If
> > > > > > > >>>>>>>>> some top-level application wants to put additional
> > > > > > > >> constraints
> > > > > > > >>> on
> > > > > > > >>>>> the
> > > > > > > >>>>>>>> data
> > > > > > > >>>>>>>>> type, like Iceberg not allowing NaN/infinity, it can
> > > still
> > > > do
> > > > > > > >>> so
> > > > > > > >>>>> one
> > > > > > > >>>>>>>> layer
> > > > > > > >>>>>>>>> above. It seems like this was the biggest point of
> > > > discussion
> > > > > > > >>>>>>> remaining.
> > > > > > > >>>>>>>>> All these options work for me, and we should move
> > forward
> > > > > > > >> with
> > > > > > > >>>> this
> > > > > > > >>>>>>>>> proposal.
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>>> Concerning additional element types like bf16, they
> can
> > > be
> > > > > > > >>>>> introduced
> > > > > > > >>>>>>> as
> > > > > > > >>>>>>>>> Parquet types (at least in the current LIST-based
> spec
> > > > > > > >>> proposal).
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>>> Concerning Will’s suggestion around representing
> VECTOR
> > > as
> > > > a
> > > > > > > >>>>> variable
> > > > > > > >>>>>>>>> length list with a MIN/MAX_LENGTH metadata
> > restriction: I
> > > > > > > >> think
> > > > > > > >>>>> that
> > > > > > > >>>>>> a
> > > > > > > >>>>>>>>> dedicated logical VECTOR type still provides enough
> > > > benefits
> > > > > > > >> by
> > > > > > > >>>>>>>> describing
> > > > > > > >>>>>>>>> the semantics of the data, and allowing to capture
> > future
> > > > > > > >>> changes
> > > > > > > >>>>>>> around
> > > > > > > >>>>>>>>> statistics. In that case, the length is better
> captured
> > > as
> > > > an
> > > > > > > >>>> exact
> > > > > > > >>>>>>>> length
> > > > > > > >>>>>>>>> for this type specifically. I’m also not sure about
> > other
> > > > use
> > > > > > > >>>> cases
> > > > > > > >>>>>> for
> > > > > > > >>>>>>>>> MIN_LENGTH, so introducing a new type constraint
> system
> > > > just
> > > > > > > >>> for
> > > > > > > >>>>> the
> > > > > > > >>>>>>>>> MIN_LENGTH=MAX_LENGTH case for vectors doesn’t have
> > > enough
> > > > > > > >>> value
> > > > > > > >>>> in
> > > > > > > >>>>>> my
> > > > > > > >>>>>>>>> view.
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>>> Best,
> > > > > > > >>>>>>>>> Philipp
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>>> On Thu, Oct 1, 2026 at 9:00 AM Will Edwards via dev <
> > > > > > > >>>>>>>>> dev@parquet.apache.org>
> > > > > > > >>>>>>>>> wrote:
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>>>> +1 Sebastian, a vector type suggests usage beyond
> > > > > > > >> normalized
> > > > > > > >>>>>>> embeddings
> > > > > > > >>>>>>>>> and
> > > > > > > >>>>>>>>>> engines really aren't going to be able to elide all
> > the
> > > > > > > >>> checks
> > > > > > > >>>>>>> anyway.
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>> Antoine, you suggested we explore metadata solutions
> > too
> > > > :)
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>> Today, a database supporting CHAR(N) or VARCHAR(n)
> > can't
> > > > > > > >>>> express
> > > > > > > >>>>>> this
> > > > > > > >>>>>>>> in
> > > > > > > >>>>>>>>>> Parquet.  Engines can write opaque blobs they
> > understand
> > > > in
> > > > > > > >>> the
> > > > > > > >>>>>>>> metadata
> > > > > > > >>>>>>>>>> (e.g. Arrow does) so the extra type info
> round-trips,
> > > but
> > > > > > > >>> this
> > > > > > > >>>>>> isn't
> > > > > > > >>>>>>>>>> standardized, portable, nor interoperable.
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>> So perhaps standardizing constraint metadata or new
> > > Thrift
> > > > > > > >>>> fields
> > > > > > > >>>>>> (my
> > > > > > > >>>>>>>>> pref)
> > > > > > > >>>>>>>>>> can generically support VECTOR(N) whilst also being
> > more
> > > > > > > >>>> broadly
> > > > > > > >>>>>>> useful
> > > > > > > >>>>>>>>> for
> > > > > > > >>>>>>>>>> other use cases and data types?
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>> Imagine metadata or Thrift fields to tag the
> > MIN_LENGTH
> > > > and
> > > > > > > >>>>>>> MAX_LENGTH
> > > > > > > >>>>>>>>> for
> > > > > > > >>>>>>>>>> Parquet's variable-length physcial types.  These
> would
> > > > > > > >> count
> > > > > > > >>>>>> logical
> > > > > > > >>>>>>>>> units
> > > > > > > >>>>>>>>>> (e.g. UTF8 code points for STRING) rather than
> bytes.
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>> Then one engine's VECTOR(N) can be written to
> Parquet
> > as
> > > > > > > >>>> LIST(N)
> > > > > > > >>>>>> with
> > > > > > > >>>>>>>>>> MIN_LENGTH = MAX_LENGTH = N and then readers - even
> > > > > > > >> different
> > > > > > > >>>>>>> engines -
> > > > > > > >>>>>>>>> can
> > > > > > > >>>>>>>>>> pick up these constraints if they support doing so.
> > We
> > > > > > > >> don't
> > > > > > > >>>>>>> actually
> > > > > > > >>>>>>>>> need
> > > > > > > >>>>>>>>>> a logical type to express a VECTOR then.
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>> This also allows standard SQL's CHAR(N) and
> VARCHAR(N)
> > > to
> > > > > > > >> be
> > > > > > > >>>>>>> expressed
> > > > > > > >>>>>>>> on
> > > > > > > >>>>>>>>>> UTF8 columns etc.
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>> This also seems generally useful in the future as we
> > > > > > > >> approach
> > > > > > > >>>>>>>> expressing
> > > > > > > >>>>>>>>>> other tensor data shapes and types.
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>> This won't change the mechanics: the storage type
> > > doesn't
> > > > > > > >>>> enforce
> > > > > > > >>>>>>> these
> > > > > > > >>>>>>>>>> constraints and engines must check them. It's
> > previously
> > > > > > > >>>>>> established
> > > > > > > >>>>>>>> that
> > > > > > > >>>>>>>>>> performance is just a software problem and there is
> no
> > > > > > > >>>>> fundamental
> > > > > > > >>>>>>>>>> performance advantage to the fixed size types in
> > > > > > > >> Dremel-like
> > > > > > > >>>>>> formats
> > > > > > > >>>>>>>> like
> > > > > > > >>>>>>>>>> Parquet.
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>> Best,
> > > > > > > >>>>>>>>>> Will
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>> On Wed, 30 Sept 2026 at 16:08, Sebastian Baunsgaard
> <
> > > > > > > >>>>>>>>> baunsgaard@apache.org
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>> wrote:
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>>> After listening in on the last meeting Wednesday, I
> > > have
> > > > > > > >>> been
> > > > > > > >>>>>>>> thinking
> > > > > > > >>>>>>>>>>> about this quite a bit, and thought I would share
> my
> > 2
> > > > > > > >>> cents.
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>> I suggest not requiring VECTOR elements to be
> finite,
> > > as
> > > > > > > >>>>> proposed
> > > > > > > >>>>>>> in
> > > > > > > >>>>>>>>> the
> > > > > > > >>>>>>>>>> PR
> > > > > > > >>>>>>>>>>> <https://github.com/apache/parquet-format/pull/624
> >.
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>> While Divjot and others note the specialization for
> > > > > > > >> UTF-8,
> > > > > > > >>>>>> Parquet
> > > > > > > >>>>>>>>>> already
> > > > > > > >>>>>>>>>>> defines DOUBLE and FLOAT as IEEE encodings
> > > > > > > >>>>>>>>>>> <
> > > > > > > >>>>>>>>
> > > > > > > >>>>
> > > > > >
> > https://github.com/apache/parquet-format/blob/master/README.md#types
> > > > > > > >>>>>>>>>> .
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>> Making NaN or +/-Inf illegal inside VECTOR would
> be a
> > > new
> > > > > > > >>>>>>> restriction
> > > > > > > >>>>>>>>> on
> > > > > > > >>>>>>>>>>> those same underlying physical types. Scalar
> > > FLOAT/DOUBLE
> > > > > > > >>>>> columns
> > > > > > > >>>>>>>> have
> > > > > > > >>>>>>>>>>> never had such a rule.
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>> It would also not match how the systems that
> actually
> > > > > > > >>> produce
> > > > > > > >>>>>> these
> > > > > > > >>>>>>>>>> vectors
> > > > > > > >>>>>>>>>>> treat floats. Most of them use IEEE floats, e.g.
> > Spark,
> > > > > > > >>>> DuckDB,
> > > > > > > >>>>>>>> NumPy,
> > > > > > > >>>>>>>>>>> PyTorch, TensorFlow, or lower level CUDA, oneMKL,
> and
> > > > > > > >>>> OpenBLAS.
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>> pgvector's "all elements must be finite" rule is an
> > > > > > > >> insert
> > > > > > > >>>>>> policy.
> > > > > > > >>>>>>>>> While
> > > > > > > >>>>>>>>>> I
> > > > > > > >>>>>>>>>>> think Parquet as a general-purpose format should
> > allow
> > > > > > > >> them
> > > > > > > >>>> out
> > > > > > > >>>>>> of
> > > > > > > >>>>>>>> the
> > > > > > > >>>>>>>>>> box.
> > > > > > > >>>>>>>>>>> It is a user system decision to disallow writing
> > these
> > > > > > > >>> values
> > > > > > > >>>>> or
> > > > > > > >>>>>> to
> > > > > > > >>>>>>>>> throw
> > > > > > > >>>>>>>>>>> errors when encountering them.
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>> A type-level ban also does not keep Inf/NaN out of
> > > vector
> > > > > > > >>>> math.
> > > > > > > >>>>>>>> Finite
> > > > > > > >>>>>>>>>>> coordinates still overflow. For instance:
> magnitude =
> > > > > > > >>>>>> ||[FLOAT_MAX,
> > > > > > > >>>>>>>>>>> FLOAT_MAX]|| where every coordinate is finite
> > overflows
> > > > > > > >> to
> > > > > > > >>>>> +Inf.
> > > > > > > >>>>>>>>> PyTorch
> > > > > > > >>>>>>>>>>> documents the same overflow: tensor([1e20,
> > > 1e20]).norm()
> > > > > > > >> is
> > > > > > > >>>> inf
> > > > > > > >>>>>> in
> > > > > > > >>>>>>>>> fp32.
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>> A generated MAGNITUDE(vec) column would then be Inf
> > > while
> > > > > > > >>> the
> > > > > > > >>>>>>> VECTOR
> > > > > > > >>>>>>>>>> itself
> > > > > > > >>>>>>>>>>> was required to be finite. The ban only forbids
> > storing
> > > > > > > >>>>> specials
> > > > > > > >>>>>> in
> > > > > > > >>>>>>>> the
> > > > > > > >>>>>>>>>>> coordinates, and does not prevent even simple
> > > downstream
> > > > > > > >>>>>>> computations
> > > > > > > >>>>>>>>>> from
> > > > > > > >>>>>>>>>>> producing them.
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>> I suggest keeping VECTOR as the type and add an
> > > optional
> > > > > > > >>> bool
> > > > > > > >>>>>>>>>>> VectorType.finite (default false). Engines that
> want
> > > > > > > >>>>>> pgvector-like
> > > > > > > >>>>>>>>>>> rejection can require the flag = true. Engines that
> > use
> > > > > > > >>>> NumPy /
> > > > > > > >>>>>>>>> PyTorch /
> > > > > > > >>>>>>>>>>> Spark arrays can keep representable IEEE values.
> For
> > > the
> > > > > > > >>> same
> > > > > > > >>>>>>>> reason, I
> > > > > > > >>>>>>>>>> do
> > > > > > > >>>>>>>>>>> not favor adding yet another type called
> > FINITE_VECTOR.
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>> Best,
> > > > > > > >>>>>>>>>>> Sebastian
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>> On Tue, Sep 29, 2026 at 2:03 PM Divjot Arora via
> dev
> > <
> > > > > > > >>>>>>>>>>> dev@parquet.apache.org>
> > > > > > > >>>>>>>>>>> wrote:
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>>> I agree with moving forward with the proposal and
> > > > > > > >>> renaming
> > > > > > > >>>>>> VECTOR
> > > > > > > >>>>>>>> to
> > > > > > > >>>>>>>>>>>> FINITE_VECTOR or something
> > > > > > > >>>>>>>>>>>> similar if others feel that better communicates
> its
> > > > > > > >>>> intent. I
> > > > > > > >>>>>> see
> > > > > > > >>>>>>>>> value
> > > > > > > >>>>>>>>>>> in
> > > > > > > >>>>>>>>>>>> having a vector type that guarantees
> > > > > > > >>>>>>>>>>>> its values are mathematically-correct vectors for
> a
> > > few
> > > > > > > >>>>>> reasons:
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>> 1. As others have mentioned, this is similar to
> the
> > > > > > > >>> STRING
> > > > > > > >>>>>>> logical
> > > > > > > >>>>>>>>> type
> > > > > > > >>>>>>>>>>>> mandating UTF-8.
> > > > > > > >>>>>>>>>>>> 2. It is not possible to rely on statistics for
> > > > > > > >> detecting
> > > > > > > >>>> NaN
> > > > > > > >>>>>> or
> > > > > > > >>>>>>>>> +-inf
> > > > > > > >>>>>>>>>>>> values as all statistics are optional.
> > > > > > > >>>>>>>>>>>> To my knowledge, there is no precedent for
> requiring
> > > > > > > >> that
> > > > > > > >>>>>> writers
> > > > > > > >>>>>>>>> emit
> > > > > > > >>>>>>>>>>>> stats for specific logical types.
> > > > > > > >>>>>>>>>>>> 3. Without statistics, NaN/inf values could flow
> > > > > > > >> through
> > > > > > > >>>> the
> > > > > > > >>>>>>> engine
> > > > > > > >>>>>>>>> and
> > > > > > > >>>>>>>>>>>> block them for specific functionality.
> > > > > > > >>>>>>>>>>>> This seems to be Lance's behavior. However, I
> don't
> > > > > > > >> think
> > > > > > > >>>>> this
> > > > > > > >>>>>> is
> > > > > > > >>>>>>>> as
> > > > > > > >>>>>>>>>> easy
> > > > > > > >>>>>>>>>>>> as it sounds and Lance has run
> > > > > > > >>>>>>>>>>>> into several bugs due to this ([1] [2] for
> > examples).
> > > > > > > >>> Their
> > > > > > > >>>>>>> Python
> > > > > > > >>>>>>>>> [3]
> > > > > > > >>>>>>>>>>> and
> > > > > > > >>>>>>>>>>>> Rust [4] APIs default to erroring on
> > > > > > > >>>>>>>>>>>> vectors that contain NaN values.
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>> I agree that Parquet is a general-purpose format,
> > but
> > > I
> > > > > > > >>>> don't
> > > > > > > >>>>>>> think
> > > > > > > >>>>>>>>>> this
> > > > > > > >>>>>>>>>>>> implies the lack of domain-specific types.
> > > > > > > >>>>>>>>>>>> Instead, this broadens the use cases where Parquet
> > > > > > > >> would
> > > > > > > >>> be
> > > > > > > >>>>>>>>> considered
> > > > > > > >>>>>>>>>> a
> > > > > > > >>>>>>>>>>>> good fit. I also fully agree with
> > > > > > > >>>>>>>>>>>> continuing work on a general fixed-size list type
> if
> > > we
> > > > > > > >>>>> believe
> > > > > > > >>>>>>>> there
> > > > > > > >>>>>>>>>> are
> > > > > > > >>>>>>>>>>>> non-vector use cases that would benefit
> > > > > > > >>>>>>>>>>>> from such guarantees at the storage layer.
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>> [1]
> https://github.com/lance-format/lance/pull/3648
> > > > > > > >>>>>>>>>>>> [2]
> > https://github.com/lance-format/lance/issues/8607
> > > > > > > >>> (see
> > > > > > > >>>>>>> point 2
> > > > > > > >>>>>>>>> in
> > > > > > > >>>>>>>>>>>> "root
> > > > > > > >>>>>>>>>>>> cause")
> > > > > > > >>>>>>>>>>>> [3]
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>>
> > > > > > > >>>>>>>
> > > > > > > >>>>>>
> > > > > > > >>>>>
> > > > > > > >>>>
> > > > > > > >>>
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/lancedb/lancedb/blob/0327618a62483feb6dca2336549845e5887cd48f/python/python/lancedb/table.py#L4807-L4911
> > > > > > > >>>>>>>>>>>> [4]
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>>
> > > > > > > >>>>>>>
> > > > > > > >>>>>>
> > > > > > > >>>>>
> > > > > > > >>>>
> > > > > > > >>>
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/lancedb/lancedb/blob/0327618a62483feb6dca2336549845e5887cd48f/rust/lancedb/src/table/add_data.rs#L105-L110
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>> On Mon, Sep 28, 2026 at 6:19 PM Philipp Fischbeck
> > via
> > > > > > > >>> dev <
> > > > > > > >>>>>>>>>>>> dev@parquet.apache.org> wrote:
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>> I also favor moving forward with the proposal as
> > is.
> > > > > > > >> I
> > > > > > > >>>> see
> > > > > > > >>>>>>> strong
> > > > > > > >>>>>>>>>> value
> > > > > > > >>>>>>>>>>>> and
> > > > > > > >>>>>>>>>>>>> use cases for the numeric vector type, and I
> agree
> > > > > > > >> with
> > > > > > > >>>> the
> > > > > > > >>>>>>>>>>> restrictions
> > > > > > > >>>>>>>>>>>>> around NaN/Inf values and element types in
> general.
> > > > > > > >>>>> Precedent
> > > > > > > >>>>>>>>> exists
> > > > > > > >>>>>>>>>> in
> > > > > > > >>>>>>>>>>>>> other types that restrict which physical type
> > values
> > > > > > > >>> are
> > > > > > > >>>>>>>> logically
> > > > > > > >>>>>>>>>>> valid
> > > > > > > >>>>>>>>>>>>> (e.g., the geospatial types needing to be
> > > > > > > >> WKB-encoded).
> > > > > > > >>>> A
> > > > > > > >>>>>>>> general
> > > > > > > >>>>>>>>>>>>> fixed-size list also has value, but I think it
> > makes
> > > > > > > >>>> sense
> > > > > > > >>>>> to
> > > > > > > >>>>>>>>>> separate
> > > > > > > >>>>>>>>>>>> the
> > > > > > > >>>>>>>>>>>>> two types. The semantics are relevant, and I can
> > see
> > > > > > > >>> how
> > > > > > > >>>>> the
> > > > > > > >>>>>>> two
> > > > > > > >>>>>>>>>> types
> > > > > > > >>>>>>>>>>>>> might diverge further in the future regarding
> > > > > > > >>> statistics
> > > > > > > >>>> or
> > > > > > > >>>>>>>>>> encodings.
> > > > > > > >>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>> I would be okay with renaming the vector type,
> but
> > it
> > > > > > > >>>> seems
> > > > > > > >>>>>>> like
> > > > > > > >>>>>>>>> the
> > > > > > > >>>>>>>>>>>> VECTOR
> > > > > > > >>>>>>>>>>>>> name is already used across other engines
> mentioned
> > > > > > > >> in
> > > > > > > >>>> this
> > > > > > > >>>>>>>> thread.
> > > > > > > >>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>> Best,
> > > > > > > >>>>>>>>>>>>> Philipp
> > > > > > > >>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>> On Sat, Sep 26, 2026 at 4:37 PM Gang Wu <
> > > > > > > >>>> ustcwg@gmail.com>
> > > > > > > >>>>>>>> wrote:
> > > > > > > >>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>> To separate orthogonal concerns, I feel we
> should
> > > > > > > >>> treat
> > > > > > > >>>>> the
> > > > > > > >>>>>>>>> general
> > > > > > > >>>>>>>>>>>>>> FixedSizeList as the underlying physical layout,
> > > > > > > >> and
> > > > > > > >>>>>> separate
> > > > > > > >>>>>>>> it
> > > > > > > >>>>>>>>>> from
> > > > > > > >>>>>>>>>>>> the
> > > > > > > >>>>>>>>>>>>>> discussion of a specialized VECTOR (or broader
> ML
> > > > > > > >>>> tensor)
> > > > > > > >>>>>>>> logical
> > > > > > > >>>>>>>>>>> type.
> > > > > > > >>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>> Currently, the immediate need is to define a
> > > > > > > >>>> specialized
> > > > > > > >>>>>>> VECTOR
> > > > > > > >>>>>>>>>> type
> > > > > > > >>>>>>>>>>> so
> > > > > > > >>>>>>>>>>>>>> that different engines and formats have a common
> > > > > > > >>>> semantic
> > > > > > > >>>>>>>>> contract
> > > > > > > >>>>>>>>>>> for
> > > > > > > >>>>>>>>>>>>>> interoperability.
> > > > > > > >>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>> If the main point of contention is just the
> > > > > > > >>>> NaN/Inf/null
> > > > > > > >>>>>>>>>> limitations,
> > > > > > > >>>>>>>>>>>>> maybe
> > > > > > > >>>>>>>>>>>>>> we can resolve this by adding a qualification to
> > > > > > > >> the
> > > > > > > >>>> name
> > > > > > > >>>>>>> (like
> > > > > > > >>>>>>>>>>> Andrew
> > > > > > > >>>>>>>>>>>>>> suggested, e.g., VECTOR_EMBEDDING or
> > FINITE_VECTOR)
> > > > > > > >>> to
> > > > > > > >>>>>> avoid
> > > > > > > >>>>>>>>>>> confusion
> > > > > > > >>>>>>>>>>>>> with
> > > > > > > >>>>>>>>>>>>>> broader ML use cases. Alternatively, we could
> > > > > > > >>>> introduce a
> > > > > > > >>>>>>>> boolean
> > > > > > > >>>>>>>>>>> flag
> > > > > > > >>>>>>>>>>>>>> (e.g., optional bool VectorType.finite) as
> > proposed
> > > > > > > >>>>>> earlier.
> > > > > > > >>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>> On Fri, Sep 25, 2026 at 3:52 AM Andrew Lamb <
> > > > > > > >>>>>>>>>> andrewlamb11@gmail.com>
> > > > > > > >>>>>>>>>>>>>> wrote:
> > > > > > > >>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>> I am a little confused about the points of
> > > > > > > >>>> disagreement
> > > > > > > >>>>>>> here.
> > > > > > > >>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>> Is it that the name `VECTOR` is too general for
> > > > > > > >> the
> > > > > > > >>>>>>> specific
> > > > > > > >>>>>>>>> use
> > > > > > > >>>>>>>>>>> case
> > > > > > > >>>>>>>>>>>>> of
> > > > > > > >>>>>>>>>>>>>>> fixed length arrays with non-finite values used
> > > > > > > >> for
> > > > > > > >>>>>> vector
> > > > > > > >>>>>>>>>>>> embeddings?
> > > > > > > >>>>>>>>>>>>>> If
> > > > > > > >>>>>>>>>>>>>>> so, maybe we could add a qualification to the
> > > > > > > >> name
> > > > > > > >>>>>>>>>> (FINITE_VECTOR?
> > > > > > > >>>>>>>>>>>>>>> FEATURE_VECTOR? VECTOR_EMBEDDING?)
> > > > > > > >>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>> It seems like there is precedent for defining
> > > > > > > >>> logical
> > > > > > > >>>>>> types
> > > > > > > >>>>>>>>> that
> > > > > > > >>>>>>>>>>>>> specify
> > > > > > > >>>>>>>>>>>>>>> certain physical types and restrict their
> > > > > > > >> contents
> > > > > > > >>> to
> > > > > > > >>>>>> help
> > > > > > > >>>>>>>>>> readers
> > > > > > > >>>>>>>>>>>>>>> interpret meaning, for example string[1] or
> JSON;
> > > > > > > >>>>>> Defining
> > > > > > > >>>>>>>> one
> > > > > > > >>>>>>>>>> for
> > > > > > > >>>>>>>>>>>>>> vector
> > > > > > > >>>>>>>>>>>>>>> embeddings seems reasonable enough to me.
> > > > > > > >>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>> Andrew
> > > > > > > >>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>> [1]:
> > > > > > > >>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>>
> > > > > > > >>>>>>>
> > > > > > > >>>>>>
> > > > > > > >>>>>
> > > > > > > >>>>
> > > > > > > >>>
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#string
> > > > > > > >>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>> On Thu, Sep 24, 2026 at 3:34 PM Daniel Weeks <
> > > > > > > >>>>>>>>> dweeks@apache.org>
> > > > > > > >>>>>>>>>>>>> wrote:
> > > > > > > >>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>> Well, Parquet is not a single-purpose format.
> > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>> Vector stores can choose whatever policy fits
> > > > > > > >>>> their
> > > > > > > >>>>>>>> usage,
> > > > > > > >>>>>>>>>> but
> > > > > > > >>>>>>>>>>>>> we're
> > > > > > > >>>>>>>>>>>>>>>>> talking about a general-purpose file format
> > > > > > > >>> that
> > > > > > > >>>>> aims
> > > > > > > >>>>>>> to
> > > > > > > >>>>>>>> be
> > > > > > > >>>>>>>>>>>> broadly
> > > > > > > >>>>>>>>>>>>>>>>> applicable.
> > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>> I agree that the format is intended to be
> > > > > > > >> general
> > > > > > > >>>>>>> purpose,
> > > > > > > >>>>>>>>> but
> > > > > > > >>>>>>>>>>>> types
> > > > > > > >>>>>>>>>>>>>>> within
> > > > > > > >>>>>>>>>>>>>>>> the format define specific representation and
> > > > > > > >>>>> semantics
> > > > > > > >>>>>>> to
> > > > > > > >>>>>>>>>>> achieve
> > > > > > > >>>>>>>>>>>>> that
> > > > > > > >>>>>>>>>>>>>>>> goal.
> > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>> "Vector" has a more general sense than ML and
> > > > > > > >>>> it's
> > > > > > > >>>>>>> quite
> > > > > > > >>>>>>>>>>> trivial
> > > > > > > >>>>>>>>>>>> to
> > > > > > > >>>>>>>>>>>>>>>>> produce vectors with non-finite values using
> > > > > > > >>> e.g.
> > > > > > > >>>>>> NumPy
> > > > > > > >>>>>>>> or
> > > > > > > >>>>>>>>>>> Pandas
> > > > > > > >>>>>>>>>>>>> (or
> > > > > > > >>>>>>>>>>>>>>>>> any other piece of dataframe or scientific
> > > > > > > >>>>> computing
> > > > > > > >>>>>>>>>> software).
> > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>> The justification for restricting non-finite
> > > > > > > >>>> values
> > > > > > > >>>>> is
> > > > > > > >>>>>>>>>> directly
> > > > > > > >>>>>>>>>>>>>> related
> > > > > > > >>>>>>>>>>>>>>> to
> > > > > > > >>>>>>>>>>>>>>>> the semantic meaning of a vector.  Vectors
> have
> > > > > > > >>>>>> specific
> > > > > > > >>>>>>>>>>>>>> characteristics
> > > > > > > >>>>>>>>>>>>>>>> (dimension, magnitude, direction, etc).
> > > > > > > >>> Non-finite
> > > > > > > >>>>>>> values
> > > > > > > >>>>>>>>> are
> > > > > > > >>>>>>>>>>>>> invalid
> > > > > > > >>>>>>>>>>>>>>> and
> > > > > > > >>>>>>>>>>>>>>>> break those characteristics.  I'd liken it to
> > > > > > > >>>> storing
> > > > > > > >>>>>>>>> non-utf8
> > > > > > > >>>>>>>>>>>> values
> > > > > > > >>>>>>>>>>>>>> in
> > > > > > > >>>>>>>>>>>>>>> a
> > > > > > > >>>>>>>>>>>>>>>> string and still calling it a string.
> > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>> That also differentiates a vector from a fixed
> > > > > > > >>>> length
> > > > > > > >>>>>>> list,
> > > > > > > >>>>>>>>>> which
> > > > > > > >>>>>>>>>>>> is
> > > > > > > >>>>>>>>>>>>>> more
> > > > > > > >>>>>>>>>>>>>>>> general and does not carry the same semantic
> > > > > > > >>>> meaning.
> > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>> I also don't think we need to bundle all
> > > > > > > >>> similarly
> > > > > > > >>>>>> shaped
> > > > > > > >>>>>>>> use
> > > > > > > >>>>>>>>>>> cases
> > > > > > > >>>>>>>>>>>>>> into
> > > > > > > >>>>>>>>>>>>>>>> the same solution (e.g. flxed size list).
> > > > > > > >> Vector
> > > > > > > >>>> is
> > > > > > > >>>>> a
> > > > > > > >>>>>>> well
> > > > > > > >>>>>>>>>>>>> established
> > > > > > > >>>>>>>>>>>>>>>> data type at this point and aligns with the
> > > > > > > >>> general
> > > > > > > >>>>>>> purpose
> > > > > > > >>>>>>>>>>> nature
> > > > > > > >>>>>>>>>>>> of
> > > > > > > >>>>>>>>>>>>>> the
> > > > > > > >>>>>>>>>>>>>>>> format (many other formats and databases
> > > > > > > >> natively
> > > > > > > >>>>>> support
> > > > > > > >>>>>>>>>> this).
> > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>> -Dan
> > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>> On Thu, Sep 24, 2026 at 11:55 AM Antoine
> > > > > > > >> Pitrou <
> > > > > > > >>>>>>>>>>>> antoine@python.org>
> > > > > > > >>>>>>>>>>>>>>>> wrote:
> > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>> Le 24/09/2026 à 20:11, Russell Spitzer a
> > > > > > > >> écrit
> > > > > > > >>> :
> > > > > > > >>>>>>>>>>>>>>>>>> I think something I haven't seen yet is the
> > > > > > > >>>> clear
> > > > > > > >>>>>>>>> use-case
> > > > > > > >>>>>>>>>>> for
> > > > > > > >>>>>>>>>>>>>>>> non-finite
> > > > > > > >>>>>>>>>>>>>>>>>> values within a vector type. There has
> > > > > > > >> been a
> > > > > > > >>>> lot
> > > > > > > >>>>>> of
> > > > > > > >>>>>>>>>>>> theoretical
> > > > > > > >>>>>>>>>>>>>>>>> discussion
> > > > > > > >>>>>>>>>>>>>>>>>> here about use cases but what toolings
> > > > > > > >>>> currently
> > > > > > > >>>>>>>> produce
> > > > > > > >>>>>>>>>>> these
> > > > > > > >>>>>>>>>>>>>>> vectors
> > > > > > > >>>>>>>>>>>>>>>>> and
> > > > > > > >>>>>>>>>>>>>>>>>> how do they function in downstream engines?
> > > > > > > >>> I'd
> > > > > > > >>>>>> like
> > > > > > > >>>>>>> to
> > > > > > > >>>>>>>>>> limit
> > > > > > > >>>>>>>>>>>>>>>> introducing
> > > > > > > >>>>>>>>>>>>>>>>>> types that have a wide range of values
> > > > > > > >> which
> > > > > > > >>>> are
> > > > > > > >>>>>>> simply
> > > > > > > >>>>>>>>> are
> > > > > > > >>>>>>>>>>>>> failure
> > > > > > > >>>>>>>>>>>>>>>> modes
> > > > > > > >>>>>>>>>>>>>>>>>> of different flavors
> > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>> "Vector" has a more general sense than ML and
> > > > > > > >>>> it's
> > > > > > > >>>>>>> quite
> > > > > > > >>>>>>>>>>> trivial
> > > > > > > >>>>>>>>>>>> to
> > > > > > > >>>>>>>>>>>>>>>>> produce vectors with non-finite values using
> > > > > > > >>> e.g.
> > > > > > > >>>>>> NumPy
> > > > > > > >>>>>>>> or
> > > > > > > >>>>>>>>>>> Pandas
> > > > > > > >>>>>>>>>>>>> (or
> > > > > > > >>>>>>>>>>>>>>>>> any other piece of dataframe or scientific
> > > > > > > >>>>> computing
> > > > > > > >>>>>>>>>> software).
> > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>> It's not like the general concept of a vector
> > > > > > > >>> is
> > > > > > > >>>> a
> > > > > > > >>>>>> IETF
> > > > > > > >>>>>>>>>>> standard
> > > > > > > >>>>>>>>>>>> or
> > > > > > > >>>>>>>>>>>>>>>>> anything, so analogies with JSON or UTF8
> > > > > > > >> (where
> > > > > > > >>>> the
> > > > > > > >>>>>>>> values
> > > > > > > >>>>>>>>>> are
> > > > > > > >>>>>>>>>>>>>> expected
> > > > > > > >>>>>>>>>>>>>>>>> to be valid according to the respective
> > > > > > > >> specs)
> > > > > > > >>> do
> > > > > > > >>>>> not
> > > > > > > >>>>>>>>> apply.
> > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>> This discussion would IMHO have a better
> > > > > > > >> chance
> > > > > > > >>>> of
> > > > > > > >>>>>>>>> converging
> > > > > > > >>>>>>>>>>> if
> > > > > > > >>>>>>>>>>>> we
> > > > > > > >>>>>>>>>>>>>>>>> separated the two orthogonal concerns of:
> > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>> 1. representing fixed-length list/vector data
> > > > > > > >>> in
> > > > > > > >>>>>>> Parquet
> > > > > > > >>>>>>>>>> (with
> > > > > > > >>>>>>>>>>> a
> > > > > > > >>>>>>>>>>>>>>>>> potential limitation to non-nested types)
> > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>> 2. representing the information that a given
> > > > > > > >>>> column
> > > > > > > >>>>>> (or
> > > > > > > >>>>>>>>>> column
> > > > > > > >>>>>>>>>>>>> chunk,
> > > > > > > >>>>>>>>>>>>>>>>> etc.) does not contain non-finite values
> > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>> ... both of which are useful on their own
> > > > > > > >>> right,
> > > > > > > >>>>> but
> > > > > > > >>>>>> do
> > > > > > > >>>>>>>> not
> > > > > > > >>>>>>>>>>> have
> > > > > > > >>>>>>>>>>>> to
> > > > > > > >>>>>>>>>>>>>> be
> > > > > > > >>>>>>>>>>>>>>>>> conflated and bundled together.
> > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>> Regards
> > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>> Antoine.
> > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>>
> > > > > > > >>>>>>>
> > > > > > > >>>>>>
> > > > > > > >>>>>
> > > > > > > >>>>
> > > > > > > >>>
> > > > > > > >>
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] Numeric Vector type: physical representation (outcome of focused call)

[Jan Finis <jp...@gmail.com>]

 assuming what will be proposed is a new logical type FINITE_VECTOR that
annotates the same column that would have been annotated as LIST before:

What I find somewhat suboptimal of FINITE_VECTOR is that finiteness is part
of the vector type. Isn't the finiteness rather a property of the contained
floats? Such a modeling would be more composable, as it would for example
also allow to have normal (non-vector) float columns that are restricted to
be finite. So, if we do think that we want finiteness in the type system, I
would rather cut it into two parts: A fixed size vector concept, and a new
way of labeling float columns finite. The vector type would then just boil
down to be a fixed size list / we could just give the list type an optional
fixed size field. Then, the column contained within it would be a float
tagged as finite.

It was mentioned that a logical type annotating a list of float would be
forward compatible, but - depending on the modeling - that is not
necessarily true: as logical types in Parquet do not support subtyping, a
column cannot be a list of float and a (finite) vector at the same time,
but only one of the two. It depends on how the type will be modeled in
thrift in the end, but if it is a new union member of `LogicalType`, then
legacy engines will not understand that such a vector can also just be
interpreted as a list of floats. They will just see a nested column type of
a logical type they don't understand, so they will likely reject reading
this column (at least that's what the engine I'm maintaining would be
doing).

In contrast, if we just give the list type an optional fixed size and float
an optional finiteness, then the resulting thrift is actually fully forward
compatible: Legacy engines will just ignore the fixed length and treat the
list as variable length and they will ignore the finiteness and treat the
element type as float.

(A side note, which most people here may already be aware of, but it's good
to repeat it: The not-NULL property doesn't need new type support at all.
The normal type system of list already allows setting the contained element
type to be optional or required. If it is set to be required, then NULLs
are not allowed.)

Cheers,
Jan





Am Di., 20. Okt. 2026 um 20:37 Uhr schrieb Russell Spitzer <
russell.spitzer@gmail.com>:

> I support proceeding with the two types. I honestly think the cost of
> adding new logical types is overestimated. If we have as many interested
> parties as have appeared in this thread we already know there is a large
> group of engineers ready to do the work and engines ready to consume (and
> produce) the type. The fact that implementations currently use heuristics,
> sampling (I loved Gunnar's talk on his hardwood work on this at CoC
> Glasgow) or custom annotations to achieve the same performance benefits
> makes me think we should just create the type and move on.
>
> On Mon, Oct 19, 2026 at 7:21 PM Ryan Blue <bl...@apache.org> wrote:
>
> > An annotation on list<float> is useful because:
> > - We want to correctly annotate what is being stored
> > - We want to unlock optimizations when each non-null list has the same
> > number of elements
> > - We want to have writer guarantees
> >
> > I don't think that logical type compatibility is a good argument. We want
> > to avoid a case where we have to treat all binary like a string. And
> > because we are introducing this in table formats after standardizing it
> > here, we can do that reliably.
> >
> > As I said in my last email, I think there's a difference between
> finiteness
> > and general "invariants" you might want. Vectors aren't vectors if they
> > don't have a length and a direction. Other invariants don't work the same
> > way. A zero-length vector is still a valid vector and can be used, just
> not
> > for certain distance metrics.
> >
> > I also want to clarify that when we looked into this more closely and
> > recognized the mismatch between the two separate use cases, I offered to
> > split out the finite vector work so that we didn't cause any headaches
> for
> > people who are interested in fixed-length list. It sounds like not
> > splitting out the use cases is causing a lot of headache. Vectors for our
> > use case need a finiteness guarantee at write time, and fixed-length
> lists
> > need to store NaN and other values.
> >
> > It's okay to have two types and we should for two different use cases.
> > Otherwise, we are forcing one to make an unnecessary compromise. I don't
> > want to force everyone to exclude NaN and infinite values from
> fixed-length
> > lists. Seems reasonable that we should also be able to have vectors that
> > exclude those values, since that is how they are used and we have to also
> > guarantee it for exchanging data with other existing databases.
> >
> > Ryan
> >
> > On Mon, Oct 19, 2026 at 3:12 AM Will Edwards via dev <
> > dev@parquet.apache.org>
> > wrote:
> >
> > > It reads well. Three small fixes:
> > >
> > > 1. **"back in May or so"** is ambiguous. The
> fixed-size-list-for-tensors
> > > framing is from Rok's opening post in May 2024. The May 2026 design doc
> > was
> > > already mostly about performance and layout. "Originally" avoids anyone
> > > correcting the date.
> > > 2. **"Spark's sparse feature vectors routinely use NaN"** isn't quite
> > > right. The NaN-for-missing behaviour comes from `VectorAssembler` and
> > > `Imputer` and applies to Spark ML vectors generally, dense or sparse.
> > > "Spark's feature vectors, dense and sparse," fixes it.
> > > 3. There are typos: "Iirc" should be "IIRC", and "Spark, BigQuery and
> > > DuckDB and pretty much…" has a doubled "and".
> > >
> > > The last sentence of the invariants paragraph could be clearer.
> "Although
> > > that doesn't mean readers shouldn't check on read" has a double
> negative
> > > and is easy to misread. "Though careful readers will still check on
> read"
> > > is simpler.
> > >
> > > With those applied:
> > >
> > > ---
> > >
> > > Hi all,
> > >
> > > The scope of this discussion keeps moving :). IIRC it originally was
> > about
> > > fixed-size lists for ML tensors, became a performance question, and is
> > now
> > > about whether vectors are mathematically valid. Each step made sense,
> > but I
> > > don't think we have consensus on where we've ended up?
> > >
> > > I think there is much useful work in what we left behind. Many ML
> vectors
> > > aren't embeddings for search. E.g. Spark's feature vectors, dense and
> > > sparse, routinely use NaN to mark missing values, and quantised formats
> > > pack codes with scales in ways Parquet can't describe today.
> ML-oriented
> > > formats like Lance are moving towards tensors, quantisation and
> indexes.
> > A
> > > type that told engines how to interpret that kind of data would enable
> > new
> > > things. An annotation on LIST<FLOAT> doesn't.
> > >
> > > My big concern with the current focus on embeddings needing a logical
> > type
> > > is compatibility. Today LIST<FLOAT> is the common format for
> embeddings,
> > > and Spark, BigQuery, DuckDB and pretty much every other library I can
> > think
> > > of offer distance functions on plain lists, so any engine can use any
> > file.
> > > If engines start requiring a VECTOR annotation, existing files and
> > anything
> > > rewritten by a tool that doesn't know the type stop working. If they
> > don't
> > > require it, the annotation adds nothing. We've seen this before with
> > > strings written without the UTF8 annotation, which engines still have
> to
> > > cope with years later.
> > >
> > > On invariants, I agree some users may want the writer to check what
> they
> > > write as they write it, but that can be a writer option rather than a
> > type.
> > > Readers that want evidence could get it from extra statistics or
> > metadata,
> > > though careful readers will still check on read.
> > >
> > > If a finite fixed-length vector contract is useful to table formats,
> > > perhaps Iceberg is the place for it, with Parquet storing plain lists.
> > I'd
> > > raise the same worry there, though: if tables start rejecting untagged
> > > columns, we've just moved the split up a layer.
> > >
> > > Best,
> > > Will
> > >
> > > On Sat, 17 Oct 2026 at 00:10, Ryan Blue <bl...@apache.org> wrote:
> > >
> > > > I agree with much of what Dan said, but I'll add some to this thread
> as
> > > > well.
> > > >
> > > > First, I want to address the arguments that were pointed out:
> > > > 1. Other vector databases have finiteness requirements
> > > > 2. "Vectors are the hot stuff right now" -- let's jump on the
> bandwagon
> > > > 3. A general solution will take too long, finiteness will make faster
> > > > progress
> > > > 4. To give a guarantee to readers that no INF/NaN/NULL will be in the
> > > > vector
> > > >
> > > > I don't think these represent our actual motivation or position,
> which
> > > > makes them function as a straw-man. This is probably not what was
> > > > intended, but I want to make sure that the misunderstanding
> > > > doesn't continue. For one thing, we aren't saying that we need finite
> > > > vectors because they are "hot stuff".
> > > >
> > > > We want to build tools that use them on top of Parquet. I think it
> > should
> > > > also be clear that we aren't advocating for finite elements blindly
> > just
> > > > because other engines do (cargo cult design!). There are real
> > > > interoperability issues if we choose to go a different direction so
> > this
> > > is
> > > > a "Chesterton's Fence" situation. It's important to understand why
> > > > those databases
> > > > made that choice. I've also pointed this out as confirmation that
> we're
> > > > going down a reasonable and established path.
> > > >
> > > > The third argument sounds like a simple misunderstanding. We're
> > pursuing
> > > a
> > > > logical type right now because that's forward compatible and we want
> to
> > > > label the data properly and set expectations for the type. Physical
> > type
> > > > changes break forward compatibility and need to be released in a new
> > > format
> > > > version . I think this is the source of the confusion since it has
> been
> > > in
> > > > the same discussion, but the choice about finiteness is separate.
> > > >
> > > > That brings us to the last argument, whether there is value in
> writers
> > > > participating in guarantees. I think there's value in adding
> > requirements
> > > > for writers. It's up to readers how they want to use those
> > requirements,
> > > > but a writer requirement allows a policy of "garbage in garbage out"
> > > rather
> > > > than being forced to handle bad data. Why would a reader need to run
> > > > validation on every element just in case a writer had a bug? And why
> > > should
> > > > the potential for writer bugs mean that it has to be the reader's
> > > problem?
> > > > If a writer produces bad data, we can absolutely reject it and tell
> the
> > > > writer to fix the bug.
> > > >
> > > > I'm also puzzled by the assertion that writer guarantees have value,
> > but
> > > > only if there are other, more specific guarantees in addition. This
> > seems
> > > > to cede the point that a contract between readers and writers makes
> > > sense,
> > > > but only if it is a _specific_ contract. But the proposed additions
> > don't
> > > > make sense in the same way.
> > > >
> > > > Take, for example, the suggestion to also disallow zero-length
> vectors.
> > > At
> > > > a glance this seems similar, but I think it assumes a specific
> context
> > > and
> > > > is better suited to be a constraint for some use cases rather than
> part
> > > of
> > > > what is considered valid for the type. That's because zero-length
> > vectors
> > > > are valid vectors with a length. You can calculate Euclidean or
> > Manhattan
> > > > distance and use them in many contexts. They don't work if you're
> > > > normalizing to a length of 1, but it's a choice to normalize and use
> a
> > > dot
> > > > product to calculate cosine distance. And at this point, we're
> talking
> > > > about the usefulness of restrictions for the type, not a Platonic
> > ideal.
> > > >
> > > > I think the restriction that makes the most sense is that vectors
> have
> > a
> > > > length and can be compared in distance calculations. 0-length vectors
> > are
> > > > valid because they point somewhere and have a length. We shouldn't
> > > restrict
> > > > uses to only cosine distance over normalized vectors. Those are use
> > case
> > > > dependent choices and there are other valid ones. I also don't think
> > that
> > > > the model used to produce a vector is part of the vector type, just
> > like
> > > > the method used to produce a metric doesn't define its type: cosine
> > > > similarity produces values between -1 and 1, but we store it in a
> > > `float`,
> > > > not a `float(cosine)`.
> > > >
> > > > Why does it make sense to have a finiteness restriction as part of
> the
> > > > type? Because that's what a (mathematical) vector is. A simple search
> > > gives
> > > > a pretty common definition: "A vector is an object that has both
> > > magnitude
> > > > and a direction". If a dimension/element is NaN, null, or infinite,
> the
> > > > vector doesn't point anywhere and it doesn't have a magnitude. Not
> only
> > > can
> > > > it not be used, it doesn't meet the most basic definition of the
> type.
> > > >
> > > > Ryan
> > > >
> > > > On Thu, Oct 15, 2026 at 1:07 AM Jan Finis <jp...@gmail.com> wrote:
> > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > Daniel,
> > > > >
> > > > > alright, so no physical optimizations intended.  We only want this
> > for
> > > > > semantic reasons, not for any significant physical optimization. If
> > > we're
> > > > > all on the same page here, that is good 👍.
> > > > >
> > > > > To phrase my remaining problem with this more concretely:
> > > > > *Let's say we add this type. What cool new stuff can we now do that
> > we
> > > > > couldn't do before?*
> > > > > That is the litmus test that a type has to pass to be useful IMHO.
> > And
> > > > here
> > > > > I can't come up with a tangible use case that wasn't possible
> before.
> > > > > Especially if we one day have fixed size lists in Parquet, the
> > > remaining
> > > > > niche for this type becomes so small, that the added complexity
> isn't
> > > > worth
> > > > > the benefit IMHO.
> > > > >
> > > > > As I guess most people on here are active maintainers of a Parquet
> > > > > implementation: Concretely, what would you be using this type for
> if
> > we
> > > > had
> > > > > it? Would you allow math only on fields tagged with this type? What
> > if
> > > > that
> > > > > math yields INF or NaN? Not allowed?
> > > > >
> > > > > Especially if we also discuss having both VECTOR and FINITE_VECTOR.
> > How
> > > > > will you guys wire this up with the type systems of your engines?
> Do
> > > your
> > > > > engines also have two types here? Do you type-decay a FINITE_VECTOR
> > to
> > > a
> > > > > VECTOR once someone does a multiplication that could yield INF? By
> > > > itself,
> > > > > a FINITE_VECTOR sounds good. Holistically looked at from a type
> > system
> > > > > perspective, it comes with a lot of complexity. Do the benefits
> > really
> > > > > outweigh the complexity here? That's why I would rather have the
> bare
> > > > > minimum: A fixed size array (list) of floats or at least just a
> > vector
> > > > > without finite. Then all these detailed questions go away.
> > > > >
> > > > > I'll take this one step further... Why is there any debate about
> this
> > > at
> > > > > > all? Why can't a type simply be added to the *specification* as
> an
> > > > > > administrative detail and be done?
> > > > >
> > > > >
> > > > > Andrew,
> > > > >
> > > > > Parquet is supposed to be a lingua franca of data. The
> specification
> > is
> > > > the
> > > > > authoritative truth for this language and all implementations are
> > > > expected
> > > > > to implement as much as possible from this spec, so anything in the
> > > spec
> > > > > does causes implementation effort. So just adding anything to this
> > spec
> > > > > without debate is not a good idea. It is even the other way around:
> > We
> > > > need
> > > > > to have way higher rigor with the spec than we need to have with
> any
> > > > > implementations. Implementations can be changed, the spec needs to
> be
> > > > > stable.
> > > > >
> > > > > A tag-only type that doesn't come with any restrictions can be
> added
> > > with
> > > > > little effort. But here we want to tie restrictions to it (no
> > NaN/Inf),
> > > > so
> > > > > we would expect implementations to enforce these. So there is some
> > > > > implementation effort expected here.
> > > > >
> > > > > Also, even if we only wanted a tag type, I would still argue that
> you
> > > > > shouldn't add those without a debate. What if A wants XML, B wants
> > > YAML,
> > > > C
> > > > > wants quaternions, D wants address, E wants phone number. If we
> just
> > > > added
> > > > > any types that are "just tags", Parquet logical types would just
> > > become a
> > > > > trash dump of any type that anyone ever thought to be useful. So
> > > rigorous
> > > > > assessment of each new type candidate is necessary.
> > > > >
> > > > > Finally, many engines couple parquet with another type system
> (e.g.,
> > > > Arrow
> > > > > or SQL). So even with a tag only type, you at least still have the
> > > > > implementation burden of wiring this up with your engine's type
> > system.
> > > > >
> > > > > So, just because logical types are "just" logical, that doesn't
> mean
> > we
> > > > can
> > > > > add them without any rigor.
> > > > >
> > > > >
> > > > > A few naive questions:
> > > > > > 1. Would a physical definition with NANs & Inf+/- allow efficient
> > > > > > encodings like ALP?
> > > > > > 2. If the physical representation allows values that make
> encodings
> > > > like
> > > > > > ALP impossible, can a tighter Logical type allow different
> > encodings?
> > > > > > 3. Can we RLE or binpack a vector?
> > > > >
> > > > >
> > > > > The simple general answer here is that all these things do not
> > require
> > > a
> > > > > logical type. Writers can already do any such optimization if the
> > > actual
> > > > > data contains no NaNs & Infs. All these physical optimizations just
> > > > require
> > > > > no actual NaNs/Infs to be present in a page, they don't need a
> > > guarantee
> > > > > that the whole file won't contain any.
> > > > >
> > > > > Cheers,
> > > > > Jan
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > Am Do., 15. Okt. 2026 um 04:28 Uhr schrieb <
> kurtiscwright@gmail.com
> > >:
> > > > >
> > > > > > Hi Everyone,
> > > > > >
> > > > > > Glad to see a Vector type being added. I have a few use cases I
> > want
> > > to
> > > > > > pursue once we have an agreed upon definition.
> > > > > >
> > > > > > In regards to that definition discussion. I feel comfortable with
> > the
> > > > > idea
> > > > > > of the semantic meaning Dan raises though I can’t shake a slight
> > > > > discomfort
> > > > > > (maybe FOMO) with limiting values in the physical definition.
> > > > > >
> > > > > > A few naive questions:
> > > > > > 1. Would a physical definition with NANs & Inf+/- allow efficient
> > > > > > encodings like ALP?
> > > > > > 2. If the physical representation allows values that make
> encodings
> > > > like
> > > > > > ALP impossible, can a tighter Logical type allow different
> > encodings?
> > > > > > 3. Can we RLE or binpack a vector?
> > > > > >
> > > > > > Best Regards,
> > > > > > Kurtis C. Wright
> > > > > > > On Oct 14, 2026, at 12:45, Daniel Weeks <dw...@apache.org>
> > wrote:
> > > > > > >
> > > > > > > Hey Jan,
> > > > > > >
> > > > > > > There's a lot that I disagree with respect to the framing of
> > these
> > > > > > > arguments and how we should be thinking about introducing
> vector
> > > as a
> > > > > > > logical type.  There's a repeated tendency to let the physical
> > > > > > > representation or performance drive discussion around types as
> > > > opposed
> > > > > to
> > > > > > > what it represents.
> > > > > > >
> > > > > > > In parquet, establishing semantic meaning is the purpose of
> > logical
> > > > > > types.
> > > > > > > The logical type separates how to interpret a value from its
> > > physical
> > > > > > > representation.  We can represent a timestamp with INT64 or
> even
> > a
> > > > > string
> > > > > > > value, so why have a type for it? Even though a
> FIXED_SIZED_LIST
> > > and
> > > > > > VECTOR
> > > > > > > may have the exact same representation and performance, they
> have
> > > > > > different
> > > > > > > semantic meaning.  Databases and engines across the ecosystem
> > > already
> > > > > > > recognize this differentiation and have separate
> representations.
> > > > The
> > > > > > > choice to separate these isn't because the discussion is too
> > > tedious;
> > > > > > > they're fundamentally different things.
> > > > > > >
> > > > > > > The finite requirement is what makes a vector a reflection of
> > what
> > > it
> > > > > > > represents.  As we've discussed in the thread, non-finite
> values
> > in
> > > > the
> > > > > > > representation violate the properties of a vector and the value
> > > > ceases
> > > > > to
> > > > > > > be a vector.  On the other hand, a 0 vector is still a vector
> > (you
> > > > can
> > > > > > > argue the utility of it, but it is a valid vector).
> > > Implementations
> > > > > can
> > > > > > > choose what they validate or how to handle execution errors;
> > that's
> > > > an
> > > > > > > implementation concern, not a requirement.  Operations
> performed
> > > on a
> > > > > > > vector depend on the context in which they are used, but the
> > > > > > representation
> > > > > > > doesn't need to solve scenarios where that application doesn't
> > make
> > > > > > sense.
> > > > > > >
> > > > > > > The guarantees or context needed to use a vector do not need to
> > be
> > > > > > inherent
> > > > > > > in the type for it to be a useful type.  We don't need the
> > currency
> > > > > used
> > > > > > > for decimal or whether a floating point is a normalized value
> as
> > > part
> > > > > of
> > > > > > > the type to make use of those types.  This conflates the use
> > cases
> > > > with
> > > > > > the
> > > > > > > type system and unnecessarily attempts to draw the context into
> > the
> > > > > type
> > > > > > > definition.  Similarly, you don't need to capture
> normalization,
> > > > model,
> > > > > > > etc. as part of the type for it to be useful.
> > > > > > >
> > > > > > > So the reason we want to add a vector type (or FINITE_VECTOR)
> is
> > > > > because
> > > > > > we
> > > > > > > want to store vectors with the semantic meaning and
> requirements
> > > > based
> > > > > on
> > > > > > > what they represent.
> > > > > > >
> > > > > > > -Dan
> > > > > > >
> > > > > > >
> > > > > > >> On Wed, Oct 14, 2026 at 1:22 AM Jan Finis <jp...@gmail.com>
> > > > wrote:
> > > > > > >>
> > > > > > >> A bit late to the party, but here we go:
> > > > > > >>
> > > > > > >> I have the strong sentiment that the FINITE_VECTOR type as it
> is
> > > > > > currently
> > > > > > >> being discussed will not be very useful and just increase
> > > > > implementation
> > > > > > >> complexity. The arguments for the type are all quite weak
> > (details
> > > > see
> > > > > > >> below). The FINITE part doesn't seem to yield any value and is
> > > not a
> > > > > > useful
> > > > > > >> restriction by itself. The vector (i.e. fixed length) part
> > should
> > > > > > rather be
> > > > > > >> generalized instead of making a one-off thing. In general, the
> > > > > existing
> > > > > > >> types are already working well for uncompressed vectors. The
> > real
> > > > > > challenge
> > > > > > >> that would bring real benefits is supporting quantization.
> This
> > is
> > > > > where
> > > > > > >> you get orders of magnitude improvements to storage space and
> > > > > processing
> > > > > > >> speed.
> > > > > > >>
> > > > > > >> Let's first discuss the FINITE part. So far, the only
> arguments
> > I
> > > > have
> > > > > > read
> > > > > > >> for this are:
> > > > > > >>
> > > > > > >> 1. many other vector databases have it
> > > > > > >> 2. vectors are the hot stuff right now, so Parquet better
> boast
> > > > having
> > > > > > >> "native" type support for vectors
> > > > > > >> 3. by focussing on this common special case, we can make rapid
> > > > > progress,
> > > > > > >> while a general solution would take too long to specify
> > > > > > >> 4. to give a guarantee to readers that no INF/NaN/NULL will be
> > in
> > > > the
> > > > > > >> vector
> > > > > > >>
> > > > > > >> the first two arguments are quite weak by themselves. Vector
> > > > databases
> > > > > > are
> > > > > > >> very specialized engines. Such engines can of course be very
> > > > > > opinionated on
> > > > > > >> their data types. A general format like Parquet must instead
> > weigh
> > > > > > whether
> > > > > > >> a special type is worth the added complexity over a more
> > general,
> > > > > widely
> > > > > > >> applicable type. In this case, it is IMHO not at all. And just
> > > > having
> > > > > a
> > > > > > >> type called "vector" for advertisement reasons to make Parquet
> > > > appear
> > > > > > more
> > > > > > >> modern seems a non-goal in the first place.
> > > > > > >>
> > > > > > >> The third "time to market" argument is valid for systems that
> > > don't
> > > > > > need a
> > > > > > >> stable interface. It is a very dangerous attitude for formats
> > like
> > > > > > Parquet
> > > > > > >> that must be stable long term and should have a consolidated,
> > > > concise,
> > > > > > >> non-redundant type system. For such long term storage formats,
> > we
> > > > need
> > > > > > to
> > > > > > >> take the time to build the right thing, as any shortcuts we
> take
> > > > will
> > > > > > stay
> > > > > > >> as blemishes in the format forever. If I read the discussion
> > > > > correctly,
> > > > > > >> this is actually the main argument as of now: We have found
> that
> > > it
> > > > is
> > > > > > hard
> > > > > > >> to align on how a fixed size list type should look like, so we
> > > want
> > > > to
> > > > > > go
> > > > > > >> the easier route of a specialized type. This seems like a
> > > > > > >> dangerous shortcut fueled by feature recency bias. Ask
> yourself:
> > > > Would
> > > > > > we
> > > > > > >> want this, if Parquet already had a fixed size list type?
> > > > > > >>
> > > > > > >> So finally, the only technical argument is the fourth one: "It
> > > > signals
> > > > > > >> semantics and ensures that no bad data lands in the format". I
> > see
> > > > two
> > > > > > big
> > > > > > >> problems with this:
> > > > > > >>
> > > > > > >> 1. Readers *still* need to test for NaN / INF because Parquet
> > > files
> > > > > may
> > > > > > be
> > > > > > >> corrupt. Nothing can stop a malicious or buggy writer from
> > > > physically
> > > > > > >> writing NaN or INF. *So you save no single line of code in a
> > > reader.
> > > > > You
> > > > > > >> save no branches, as you still need to handle the INF/NaN
> case.*
> > > So
> > > > > this
> > > > > > >> doesn't help with performance at all. The only difference is,
> > that
> > > > the
> > > > > > >> error message changes from "found an invalid INF value in the
> > > data"
> > > > to
> > > > > > >> "Your parquet file seems to be corrupt". It is actually quite
> > the
> > > > > > opposite,
> > > > > > >> readers and writers get more complex having to handle this new
> > > type.
> > > > > > >>
> > > > > > >> If you now argue that it's not about performance, it's about
> > > defined
> > > > > > >> semantics and erroring out early in writers, then the second
> > > problem
> > > > > > >> arises:
> > > > > > >>
> > > > > > >> 2. The proposed no-NaN/INF/NULL guarantee is just one of
> > multiple
> > > > > > >> guarantees that you need for sensible vector processing. If
> "no
> > > > > > >> INF/NaN/NULL" was the only guarantee you needed, I would agree
> > > that
> > > > > > >> FINITE_VECTOR is useful, but it is by no means the only
> > guarantee
> > > > you
> > > > > > need,
> > > > > > >> instead, depending on the distance metric, you also need the
> > > > following
> > > > > > >> guarantees. E.g.:
> > > > > > >>
> > > > > > >>   - No vector may be the 0 vector, otherwise, e.g., cosine
> > > > similarity
> > > > > is
> > > > > > >>   undefined. Cosine similarity is often used as default, so
> this
> > > is
> > > > > > >> actually
> > > > > > >>   quite important in practice.
> > > > > > >>   - If we want to use a dot product, then all vectors need to
> > > > > guarantee
> > > > > > a
> > > > > > >>   length of 1. This would actually be quite useful for
> > performance
> > > > > > >> reasons.
> > > > > > >>   - The vectors must come from the same model as the vector
> used
> > > for
> > > > > the
> > > > > > >>   search. Otherwise, the search yields a numeric result, but
> > that
> > > > > result
> > > > > > >> has
> > > > > > >>   no sensible meaning.
> > > > > > >>
> > > > > > >> So, what use is a type that just guarantees part of the things
> > you
> > > > > need
> > > > > > for
> > > > > > >> vector search but not others? You still need additional
> > guarantees
> > > > in
> > > > > > >> addition to that type. What makes INF/NaN so important in
> > > comparison
> > > > > to,
> > > > > > >> say, "no 0 vector present", that it warrants an own type,
> while
> > > the
> > > > > > other
> > > > > > >> guarantees do not? If we really want this type as a basis for
> > > > > > cross-engine
> > > > > > >> sharing of vectors, then it needs to give *all* guarantees.
> > > > > > >>
> > > > > > >> I'm not against a specialized vector type if it really carries
> > > *all*
> > > > > > >> information to be useful by itself. If it carried all that
> > > > > information,
> > > > > > >> then a reading engine could indeed validate that only
> meaningful
> > > > > vector
> > > > > > >> searches are executed. But as long as it is just a half-baked
> > > thing,
> > > > > it
> > > > > > is
> > > > > > >> not useful by itself and just a burden on each implementer and
> > > > > > maintainer.
> > > > > > >> If we need extra information anyway, why not just stick to a
> > > general
> > > > > > type
> > > > > > >> like VECTOR (without FINITE).
> > > > > > >>
> > > > > > >> If you're a proponent of FINITE_VECTOR, ask yourself (or tell
> > us)
> > > > what
> > > > > > >> advantage you will *really* get over just a (potentially fixed
> > > size)
> > > > > > List
> > > > > > >> of Float? As discussed, it's not performance. It's also not a
> > > > semantic
> > > > > > >> marker that is sufficient by itself. It's just a small hint
> that
> > > > > allows
> > > > > > >> writers to error out once they see a NaN or INF in the input,
> > > while
> > > > > they
> > > > > > >> will still happily write a bogus 0 vector. This seems to be of
> > > zero
> > > > > > >> benefit.
> > > > > > >>
> > > > > > >> Anecdotally, we currently handle petabytes of vectors in
> > > production
> > > > in
> > > > > > >> Data360. We simply store them in a List of Float, with an
> > > optimized
> > > > > > R-Level
> > > > > > >> reader implementation that rapidly checks that all vectors are
> > the
> > > > > same
> > > > > > >> length. This solution works just fine. The overhead of R-level
> > > > > decoding
> > > > > > is
> > > > > > >> barely measurable. The check is super fast and R-levels are
> tiny
> > > in
> > > > > > >> comparison to the payload vectors, so also the added I/O is
> > > > > negligible.
> > > > > > >> This is also why I'm so firm in my sentiment: We are running a
> > > > > > production
> > > > > > >> grade system processing trillions of vectors right now and see
> > no
> > > > big
> > > > > > >> issues with the types that Parquet currently provides.
> > > > > > >>
> > > > > > >> As I don't just want to be a naysayer and point out problems
> > with
> > > > the
> > > > > > >> current design, here are the directions that I would propose
> > > indeed:
> > > > > > >>
> > > > > > >>   - a general fixed size list type. Don't do shortcuts just
> > > because
> > > > > the
> > > > > > >>   discussion is too tedious.
> > > > > > >>   - a vector type that indeed carries enough information to be
> > > > really
> > > > > > >>   useful for vector search by itself. (e.g., model name /
> > version,
> > > > > > >> optional
> > > > > > >>   guarantees for no 0 vector / normalized vector length). With
> > > such
> > > > a
> > > > > > >> type,
> > > > > > >>   you could really do well defined vector searches even if the
> > > > writing
> > > > > > >> engine
> > > > > > >>   is completely different to the reading one.
> > > > > > >>   - Standardize quantizations. This will be a real game
> changer
> > > for
> > > > > > >>   performance.
> > > > > > >>
> > > > > > >> Cheers,
> > > > > > >> Jan
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >>> Am Mi., 14. Okt. 2026 um 03:36 Uhr schrieb Gang Wu <
> > > > ustcwg@gmail.com
> > > > > >:
> > > > > > >>>
> > > > > > >>> I think separating them into FINITE_VECTOR and
> > FIXED_LENGTH_LIST
> > > > is a
> > > > > > >> good
> > > > > > >>> idea, though I think renaming FINITE_VECTOR to VECTOR looks
> > more
> > > > > > natural.
> > > > > > >>> Should we move forward with proposing FINITE_VECTOR/VECTOR
> > first?
> > > > > > >>>
> > > > > > >>> On Tue, Oct 13, 2026 at 6:26 PM Alkis Evlogimenos via dev <
> > > > > > >>> dev@parquet.apache.org> wrote:
> > > > > > >>>
> > > > > > >>>> I suggest we go with VECTOR for the finite vector to keep
> the
> > > name
> > > > > the
> > > > > > >>> same
> > > > > > >>>> as other engines/type systems. If we want a non-finite one
> in
> > > the
> > > > > > >> future
> > > > > > >>> we
> > > > > > >>>> can name that something else or we add fixed size list that
> > will
> > > > > serve
> > > > > > >>> the
> > > > > > >>>> same purpose without a special name.
> > > > > > >>>>
> > > > > > >>>>
> > > > > > >>>> On Thu, Oct 8, 2026 at 10:20 PM Ryan Blue <bl...@apache.org>
> > > > wrote:
> > > > > > >>>>
> > > > > > >>>>> A logical type would:
> > > > > > >>>>> - Describe data for users and readers to understand what
> the
> > > file
> > > > > > >>>> contains
> > > > > > >>>>> - Signal that repetition and definition levels will
> reflect a
> > > > fixed
> > > > > > >>>> number
> > > > > > >>>>> of elements, which implementations can use to optimize
> > decoding
> > > > > > >>>>> - Guarantee that there are no null elements (because the
> type
> > > > > cannot
> > > > > > >> be
> > > > > > >>>>> applied to a list with optional elements)
> > > > > > >>>>> - [FINITE_VECTOR] Guarantee that no elements are NaN or
> > > Infinity
> > > > > > >> values
> > > > > > >>>> so
> > > > > > >>>>> that the vectors have a finite length and are suitable for
> > > > distance
> > > > > > >>>>> calculations
> > > > > > >>>>>
> > > > > > >>>>> The finite value restriction is a restriction on writers to
> > > > prevent
> > > > > > >> bad
> > > > > > >>>>> data from being written.
> > > > > > >>>>>
> > > > > > >>>>> Ignoring logical types that are not understood is a rule
> for
> > > > > > >> readers. A
> > > > > > >>>>> reader can handle vectors and pass it on to another layer.
> > For
> > > > > > >>> instance,
> > > > > > >>>> a
> > > > > > >>>>> Parquet reader that doesn't support geometry passes values
> as
> > > > > > >> `binary`.
> > > > > > >>>>> That binary value can be passed to a geometry UDF, even if
> > the
> > > > > > >>> underlying
> > > > > > >>>>> engine and Parquet reader doesn't have support.
> > > > > > >>>>>
> > > > > > >>>>> A tool that doesn't understand FINITE_VECTOR can similarly
> > > pass a
> > > > > > >> list
> > > > > > >>> of
> > > > > > >>>>> floats along to a UDF. But it is still important to the
> user
> > > that
> > > > > bad
> > > > > > >>>>> values are not present in the data. In your example, bad
> > values
> > > > are
> > > > > > >>>> vectors
> > > > > > >>>>> that cause the centroid calculation to fail, like vectors
> > with
> > > > > > >> infinite
> > > > > > >>>>> length.
> > > > > > >>>>>
> > > > > > >>>>> Ryan
> > > > > > >>>>>
> > > > > > >>>>> On Thu, Oct 8, 2026 at 12:54 PM Will Edwards via dev <
> > > > > > >>>>> dev@parquet.apache.org>
> > > > > > >>>>> wrote:
> > > > > > >>>>>
> > > > > > >>>>>> Could we clarify the use cases for a VECTOR annotation on
> > > > > > >>> LIST<FLOAT>?
> > > > > > >>>>>> Specifically, what does it make possible that's impossible
> > > > today,
> > > > > > >> and
> > > > > > >>>>> what
> > > > > > >>>>>> does it make easier that's currently hard?
> > > > > > >>>>>>
> > > > > > >>>>>> I'm struggling to find a strong answer. I'd like every
> > engine
> > > to
> > > > > > >>> offer
> > > > > > >>>>>> vector functions on lists of floats without needing a new
> > > > logical
> > > > > > >>> type.
> > > > > > >>>>>>
> > > > > > >>>>>> To be clear, I'm still enthusiastic about a vector type
> that
> > > > > > >>> describes
> > > > > > >>>>> how
> > > > > > >>>>>> to unpack and scale composite representations (quantised
> > codes
> > > > > plus
> > > > > > >>>>> scales,
> > > > > > >>>>>> codebooks, etc.), in the direction ML is going. That adds
> > > > > > >> information
> > > > > > >>>>>> engines don't have. A label on LIST<FLOAT> doesn't:
> engines
> > > > > already
> > > > > > >>>>>> understand the physical type, and many already offer
> vector
> > > > > > >> functions
> > > > > > >>>> on
> > > > > > >>>>>> it. I don't see a performance benefit, and NaN/null/Inf
> > checks
> > > > are
> > > > > > >>>> cheap
> > > > > > >>>>>> enough that engines should do them anyway. The label also
> > > > doesn't
> > > > > > >> say
> > > > > > >>>>> what
> > > > > > >>>>>> you can do with the column. Knowing that a column holds
> > > > embeddings
> > > > > > >>>>> doesn't
> > > > > > >>>>>> say how they can be used, just as zipcode and age are both
> > > > > integers
> > > > > > >>> but
> > > > > > >>>>>> multiplying them is meaningless.
> > > > > > >>>>>>
> > > > > > >>>>>> My concern is the opposite effect. We're establishing that
> > > > engines
> > > > > > >>>> should
> > > > > > >>>>>> ignore logical types they don't understand. So a tool that
> > > > doesn't
> > > > > > >>> know
> > > > > > >>>>>> VECTOR will read the column as LIST<FLOAT>, let the user
> > > compute
> > > > > > >>>>> centroids
> > > > > > >>>>>> (say), and write plain LIST<FLOAT> back out. If other
> > engines
> > > > > > >>> restrict
> > > > > > >>>>>> embedding functions to VECTOR-annotated columns, that file
> > > stops
> > > > > > >>>> working
> > > > > > >>>>>> with them, through no fault of the user. An annotation
> that
> > > > > doesn't
> > > > > > >>>>> change
> > > > > > >>>>>> what the data is shouldn't be able to make it unusable.
> > > > > > >>>>>>
> > > > > > >>>>>> Am I missing a use case?
> > > > > > >>>>>>
> > > > > > >>>>>> Best, Will
> > > > > > >>>>>>
> > > > > > >>>>>> On Thu, 8 Oct 2026 at 19:53, Steven Wu <
> > stevenz3wu@gmail.com>
> > > > > > >> wrote:
> > > > > > >>>>>>
> > > > > > >>>>>>> I also think a FINITE_VECTOR type brings a lot of values
> to
> > > > many
> > > > > > >>> use
> > > > > > >>>>>> cases.
> > > > > > >>>>>>> It is would be great to make progress on the
> FINITE_VECTOR,
> > > and
> > > > > > >> not
> > > > > > >>>> to
> > > > > > >>>>>> fit
> > > > > > >>>>>>> every use cases into the FIXED_LENGTH_LIST.
> > > > > > >>>>>>>
> > > > > > >>>>>>> On Mon, Oct 5, 2026 at 1:26 PM Ryan Blue <
> blue@apache.org>
> > > > > > >> wrote:
> > > > > > >>>>>>>
> > > > > > >>>>>>>> Going through this thread again, it looks like there
> are a
> > > few
> > > > > > >>>> things
> > > > > > >>>>>>> that
> > > > > > >>>>>>>> people generally agree on:
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> - In the short term, we want to introduce a logical type
> > or
> > > > > > >>> types;
> > > > > > >>>> a
> > > > > > >>>>>>> better
> > > > > > >>>>>>>> physical representation can be done later. We can focus
> on
> > > > > > >>> logical
> > > > > > >>>>>> types.
> > > > > > >>>>>>>> - There is precedent in logical types for restricting
> > > values,
> > > > > > >>> like
> > > > > > >>>>> UTF8
> > > > > > >>>>>>>> strings to contain only UTF-8 codepoints. I think we
> also
> > > > agree
> > > > > > >>>> that
> > > > > > >>>>> we
> > > > > > >>>>>>>> could restrict vector elements to non-nested or
> primitive
> > > > > > >> types.
> > > > > > >>>>>>>> - There are many examples of databases that restrict
> their
> > > > > > >> vector
> > > > > > >>>>> types
> > > > > > >>>>>>> to
> > > > > > >>>>>>>> non-null, finite elements.
> > > > > > >>>>>>>> - Some people want to have write-time guarantees that
> > > vectors
> > > > > > >>> have
> > > > > > >>>>> only
> > > > > > >>>>>>>> non-null, finite elements
> > > > > > >>>>>>>> - Some people want to extend this work to include
> tensors,
> > > > > > >>> shorter
> > > > > > >>>>>>> floating
> > > > > > >>>>>>>> point representations, variable lengths, NaN or Infinity
> > > > > > >>> elements,
> > > > > > >>>>> etc.
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> It sounds like we are nearly there, but have two halves
> of
> > > the
> > > > > > >>>>>> community
> > > > > > >>>>>>>> that care about different things. That just means we
> > should
> > > > > > >> have
> > > > > > >>>>>> multiple
> > > > > > >>>>>>>> types. It's entirely reasonable to want a type that
> allows
> > > NaN
> > > > > > >>>>> values,
> > > > > > >>>>>>> and
> > > > > > >>>>>>>> it's also reasonable for those of us building databases,
> > > table
> > > > > > >>>>>> standards,
> > > > > > >>>>>>>> and vector indexes to care about guarantees that
> elements
> > > are
> > > > > > >>>> finite.
> > > > > > >>>>>>>> There's more than enough examples to show that this is a
> > > > > > >>> legitimate
> > > > > > >>>>> use
> > > > > > >>>>>>>> case.
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> Since I don't see the concerns of the different use
> cases
> > > > > > >>> changing,
> > > > > > >>>>> it
> > > > > > >>>>>>>> looks like we need to find a compromise to move
> forward. I
> > > > > > >>> propose
> > > > > > >>>>> that
> > > > > > >>>>>>> we
> > > > > > >>>>>>>> introduce a FINITE_VECTOR type and separately a
> > > > > > >> FIXED_LENGTH_LIST
> > > > > > >>>> or
> > > > > > >>>>>>>> another VECTOR type that allows infinite and NaN values.
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> Can we agree on a compromise? Is there a strong reason
> to
> > > try
> > > > > > >> to
> > > > > > >>>> fit
> > > > > > >>>>>>> these
> > > > > > >>>>>>>> use cases into a single type? What if that means not
> > making
> > > > > > >>>> progress?
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> Ryan
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> On Mon, Oct 5, 2026 at 12:15 PM Philipp Fischbeck via
> dev
> > <
> > > > > > >>>>>>>> dev@parquet.apache.org> wrote:
> > > > > > >>>>>>>>
> > > > > > >>>>>>>>> Thinking about this more, I am also fine with pulling
> out
> > > the
> > > > > > >>>>>> “NaN/Inf”
> > > > > > >>>>>>>>> requirement. Either into a type boolean (my
> preference),
> > or
> > > > > > >>>> putting
> > > > > > >>>>>> no
> > > > > > >>>>>>>>> restrictions on the Parquet level, allowing all IEEE
> 754
> > > > > > >> float
> > > > > > >>>>>> numbers.
> > > > > > >>>>>>>> If
> > > > > > >>>>>>>>> some top-level application wants to put additional
> > > > > > >> constraints
> > > > > > >>> on
> > > > > > >>>>> the
> > > > > > >>>>>>>> data
> > > > > > >>>>>>>>> type, like Iceberg not allowing NaN/infinity, it can
> > still
> > > do
> > > > > > >>> so
> > > > > > >>>>> one
> > > > > > >>>>>>>> layer
> > > > > > >>>>>>>>> above. It seems like this was the biggest point of
> > > discussion
> > > > > > >>>>>>> remaining.
> > > > > > >>>>>>>>> All these options work for me, and we should move
> forward
> > > > > > >> with
> > > > > > >>>> this
> > > > > > >>>>>>>>> proposal.
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> Concerning additional element types like bf16, they can
> > be
> > > > > > >>>>> introduced
> > > > > > >>>>>>> as
> > > > > > >>>>>>>>> Parquet types (at least in the current LIST-based spec
> > > > > > >>> proposal).
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> Concerning Will’s suggestion around representing VECTOR
> > as
> > > a
> > > > > > >>>>> variable
> > > > > > >>>>>>>>> length list with a MIN/MAX_LENGTH metadata
> restriction: I
> > > > > > >> think
> > > > > > >>>>> that
> > > > > > >>>>>> a
> > > > > > >>>>>>>>> dedicated logical VECTOR type still provides enough
> > > benefits
> > > > > > >> by
> > > > > > >>>>>>>> describing
> > > > > > >>>>>>>>> the semantics of the data, and allowing to capture
> future
> > > > > > >>> changes
> > > > > > >>>>>>> around
> > > > > > >>>>>>>>> statistics. In that case, the length is better captured
> > as
> > > an
> > > > > > >>>> exact
> > > > > > >>>>>>>> length
> > > > > > >>>>>>>>> for this type specifically. I’m also not sure about
> other
> > > use
> > > > > > >>>> cases
> > > > > > >>>>>> for
> > > > > > >>>>>>>>> MIN_LENGTH, so introducing a new type constraint system
> > > just
> > > > > > >>> for
> > > > > > >>>>> the
> > > > > > >>>>>>>>> MIN_LENGTH=MAX_LENGTH case for vectors doesn’t have
> > enough
> > > > > > >>> value
> > > > > > >>>> in
> > > > > > >>>>>> my
> > > > > > >>>>>>>>> view.
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> Best,
> > > > > > >>>>>>>>> Philipp
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> On Thu, Oct 1, 2026 at 9:00 AM Will Edwards via dev <
> > > > > > >>>>>>>>> dev@parquet.apache.org>
> > > > > > >>>>>>>>> wrote:
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>>> +1 Sebastian, a vector type suggests usage beyond
> > > > > > >> normalized
> > > > > > >>>>>>> embeddings
> > > > > > >>>>>>>>> and
> > > > > > >>>>>>>>>> engines really aren't going to be able to elide all
> the
> > > > > > >>> checks
> > > > > > >>>>>>> anyway.
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>> Antoine, you suggested we explore metadata solutions
> too
> > > :)
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>> Today, a database supporting CHAR(N) or VARCHAR(n)
> can't
> > > > > > >>>> express
> > > > > > >>>>>> this
> > > > > > >>>>>>>> in
> > > > > > >>>>>>>>>> Parquet.  Engines can write opaque blobs they
> understand
> > > in
> > > > > > >>> the
> > > > > > >>>>>>>> metadata
> > > > > > >>>>>>>>>> (e.g. Arrow does) so the extra type info round-trips,
> > but
> > > > > > >>> this
> > > > > > >>>>>> isn't
> > > > > > >>>>>>>>>> standardized, portable, nor interoperable.
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>> So perhaps standardizing constraint metadata or new
> > Thrift
> > > > > > >>>> fields
> > > > > > >>>>>> (my
> > > > > > >>>>>>>>> pref)
> > > > > > >>>>>>>>>> can generically support VECTOR(N) whilst also being
> more
> > > > > > >>>> broadly
> > > > > > >>>>>>> useful
> > > > > > >>>>>>>>> for
> > > > > > >>>>>>>>>> other use cases and data types?
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>> Imagine metadata or Thrift fields to tag the
> MIN_LENGTH
> > > and
> > > > > > >>>>>>> MAX_LENGTH
> > > > > > >>>>>>>>> for
> > > > > > >>>>>>>>>> Parquet's variable-length physcial types.  These would
> > > > > > >> count
> > > > > > >>>>>> logical
> > > > > > >>>>>>>>> units
> > > > > > >>>>>>>>>> (e.g. UTF8 code points for STRING) rather than bytes.
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>> Then one engine's VECTOR(N) can be written to Parquet
> as
> > > > > > >>>> LIST(N)
> > > > > > >>>>>> with
> > > > > > >>>>>>>>>> MIN_LENGTH = MAX_LENGTH = N and then readers - even
> > > > > > >> different
> > > > > > >>>>>>> engines -
> > > > > > >>>>>>>>> can
> > > > > > >>>>>>>>>> pick up these constraints if they support doing so.
> We
> > > > > > >> don't
> > > > > > >>>>>>> actually
> > > > > > >>>>>>>>> need
> > > > > > >>>>>>>>>> a logical type to express a VECTOR then.
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>> This also allows standard SQL's CHAR(N) and VARCHAR(N)
> > to
> > > > > > >> be
> > > > > > >>>>>>> expressed
> > > > > > >>>>>>>> on
> > > > > > >>>>>>>>>> UTF8 columns etc.
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>> This also seems generally useful in the future as we
> > > > > > >> approach
> > > > > > >>>>>>>> expressing
> > > > > > >>>>>>>>>> other tensor data shapes and types.
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>> This won't change the mechanics: the storage type
> > doesn't
> > > > > > >>>> enforce
> > > > > > >>>>>>> these
> > > > > > >>>>>>>>>> constraints and engines must check them. It's
> previously
> > > > > > >>>>>> established
> > > > > > >>>>>>>> that
> > > > > > >>>>>>>>>> performance is just a software problem and there is no
> > > > > > >>>>> fundamental
> > > > > > >>>>>>>>>> performance advantage to the fixed size types in
> > > > > > >> Dremel-like
> > > > > > >>>>>> formats
> > > > > > >>>>>>>> like
> > > > > > >>>>>>>>>> Parquet.
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>> Best,
> > > > > > >>>>>>>>>> Will
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>> On Wed, 30 Sept 2026 at 16:08, Sebastian Baunsgaard <
> > > > > > >>>>>>>>> baunsgaard@apache.org
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>> wrote:
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>>> After listening in on the last meeting Wednesday, I
> > have
> > > > > > >>> been
> > > > > > >>>>>>>> thinking
> > > > > > >>>>>>>>>>> about this quite a bit, and thought I would share my
> 2
> > > > > > >>> cents.
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>> I suggest not requiring VECTOR elements to be finite,
> > as
> > > > > > >>>>> proposed
> > > > > > >>>>>>> in
> > > > > > >>>>>>>>> the
> > > > > > >>>>>>>>>> PR
> > > > > > >>>>>>>>>>> <https://github.com/apache/parquet-format/pull/624>.
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>> While Divjot and others note the specialization for
> > > > > > >> UTF-8,
> > > > > > >>>>>> Parquet
> > > > > > >>>>>>>>>> already
> > > > > > >>>>>>>>>>> defines DOUBLE and FLOAT as IEEE encodings
> > > > > > >>>>>>>>>>> <
> > > > > > >>>>>>>>
> > > > > > >>>>
> > > > >
> https://github.com/apache/parquet-format/blob/master/README.md#types
> > > > > > >>>>>>>>>> .
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>> Making NaN or +/-Inf illegal inside VECTOR would be a
> > new
> > > > > > >>>>>>> restriction
> > > > > > >>>>>>>>> on
> > > > > > >>>>>>>>>>> those same underlying physical types. Scalar
> > FLOAT/DOUBLE
> > > > > > >>>>> columns
> > > > > > >>>>>>>> have
> > > > > > >>>>>>>>>>> never had such a rule.
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>> It would also not match how the systems that actually
> > > > > > >>> produce
> > > > > > >>>>>> these
> > > > > > >>>>>>>>>> vectors
> > > > > > >>>>>>>>>>> treat floats. Most of them use IEEE floats, e.g.
> Spark,
> > > > > > >>>> DuckDB,
> > > > > > >>>>>>>> NumPy,
> > > > > > >>>>>>>>>>> PyTorch, TensorFlow, or lower level CUDA, oneMKL, and
> > > > > > >>>> OpenBLAS.
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>> pgvector's "all elements must be finite" rule is an
> > > > > > >> insert
> > > > > > >>>>>> policy.
> > > > > > >>>>>>>>> While
> > > > > > >>>>>>>>>> I
> > > > > > >>>>>>>>>>> think Parquet as a general-purpose format should
> allow
> > > > > > >> them
> > > > > > >>>> out
> > > > > > >>>>>> of
> > > > > > >>>>>>>> the
> > > > > > >>>>>>>>>> box.
> > > > > > >>>>>>>>>>> It is a user system decision to disallow writing
> these
> > > > > > >>> values
> > > > > > >>>>> or
> > > > > > >>>>>> to
> > > > > > >>>>>>>>> throw
> > > > > > >>>>>>>>>>> errors when encountering them.
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>> A type-level ban also does not keep Inf/NaN out of
> > vector
> > > > > > >>>> math.
> > > > > > >>>>>>>> Finite
> > > > > > >>>>>>>>>>> coordinates still overflow. For instance: magnitude =
> > > > > > >>>>>> ||[FLOAT_MAX,
> > > > > > >>>>>>>>>>> FLOAT_MAX]|| where every coordinate is finite
> overflows
> > > > > > >> to
> > > > > > >>>>> +Inf.
> > > > > > >>>>>>>>> PyTorch
> > > > > > >>>>>>>>>>> documents the same overflow: tensor([1e20,
> > 1e20]).norm()
> > > > > > >> is
> > > > > > >>>> inf
> > > > > > >>>>>> in
> > > > > > >>>>>>>>> fp32.
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>> A generated MAGNITUDE(vec) column would then be Inf
> > while
> > > > > > >>> the
> > > > > > >>>>>>> VECTOR
> > > > > > >>>>>>>>>> itself
> > > > > > >>>>>>>>>>> was required to be finite. The ban only forbids
> storing
> > > > > > >>>>> specials
> > > > > > >>>>>> in
> > > > > > >>>>>>>> the
> > > > > > >>>>>>>>>>> coordinates, and does not prevent even simple
> > downstream
> > > > > > >>>>>>> computations
> > > > > > >>>>>>>>>> from
> > > > > > >>>>>>>>>>> producing them.
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>> I suggest keeping VECTOR as the type and add an
> > optional
> > > > > > >>> bool
> > > > > > >>>>>>>>>>> VectorType.finite (default false). Engines that want
> > > > > > >>>>>> pgvector-like
> > > > > > >>>>>>>>>>> rejection can require the flag = true. Engines that
> use
> > > > > > >>>> NumPy /
> > > > > > >>>>>>>>> PyTorch /
> > > > > > >>>>>>>>>>> Spark arrays can keep representable IEEE values. For
> > the
> > > > > > >>> same
> > > > > > >>>>>>>> reason, I
> > > > > > >>>>>>>>>> do
> > > > > > >>>>>>>>>>> not favor adding yet another type called
> FINITE_VECTOR.
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>> Best,
> > > > > > >>>>>>>>>>> Sebastian
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>> On Tue, Sep 29, 2026 at 2:03 PM Divjot Arora via dev
> <
> > > > > > >>>>>>>>>>> dev@parquet.apache.org>
> > > > > > >>>>>>>>>>> wrote:
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>>> I agree with moving forward with the proposal and
> > > > > > >>> renaming
> > > > > > >>>>>> VECTOR
> > > > > > >>>>>>>> to
> > > > > > >>>>>>>>>>>> FINITE_VECTOR or something
> > > > > > >>>>>>>>>>>> similar if others feel that better communicates its
> > > > > > >>>> intent. I
> > > > > > >>>>>> see
> > > > > > >>>>>>>>> value
> > > > > > >>>>>>>>>>> in
> > > > > > >>>>>>>>>>>> having a vector type that guarantees
> > > > > > >>>>>>>>>>>> its values are mathematically-correct vectors for a
> > few
> > > > > > >>>>>> reasons:
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>> 1. As others have mentioned, this is similar to the
> > > > > > >>> STRING
> > > > > > >>>>>>> logical
> > > > > > >>>>>>>>> type
> > > > > > >>>>>>>>>>>> mandating UTF-8.
> > > > > > >>>>>>>>>>>> 2. It is not possible to rely on statistics for
> > > > > > >> detecting
> > > > > > >>>> NaN
> > > > > > >>>>>> or
> > > > > > >>>>>>>>> +-inf
> > > > > > >>>>>>>>>>>> values as all statistics are optional.
> > > > > > >>>>>>>>>>>> To my knowledge, there is no precedent for requiring
> > > > > > >> that
> > > > > > >>>>>> writers
> > > > > > >>>>>>>>> emit
> > > > > > >>>>>>>>>>>> stats for specific logical types.
> > > > > > >>>>>>>>>>>> 3. Without statistics, NaN/inf values could flow
> > > > > > >> through
> > > > > > >>>> the
> > > > > > >>>>>>> engine
> > > > > > >>>>>>>>> and
> > > > > > >>>>>>>>>>>> block them for specific functionality.
> > > > > > >>>>>>>>>>>> This seems to be Lance's behavior. However, I don't
> > > > > > >> think
> > > > > > >>>>> this
> > > > > > >>>>>> is
> > > > > > >>>>>>>> as
> > > > > > >>>>>>>>>> easy
> > > > > > >>>>>>>>>>>> as it sounds and Lance has run
> > > > > > >>>>>>>>>>>> into several bugs due to this ([1] [2] for
> examples).
> > > > > > >>> Their
> > > > > > >>>>>>> Python
> > > > > > >>>>>>>>> [3]
> > > > > > >>>>>>>>>>> and
> > > > > > >>>>>>>>>>>> Rust [4] APIs default to erroring on
> > > > > > >>>>>>>>>>>> vectors that contain NaN values.
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>> I agree that Parquet is a general-purpose format,
> but
> > I
> > > > > > >>>> don't
> > > > > > >>>>>>> think
> > > > > > >>>>>>>>>> this
> > > > > > >>>>>>>>>>>> implies the lack of domain-specific types.
> > > > > > >>>>>>>>>>>> Instead, this broadens the use cases where Parquet
> > > > > > >> would
> > > > > > >>> be
> > > > > > >>>>>>>>> considered
> > > > > > >>>>>>>>>> a
> > > > > > >>>>>>>>>>>> good fit. I also fully agree with
> > > > > > >>>>>>>>>>>> continuing work on a general fixed-size list type if
> > we
> > > > > > >>>>> believe
> > > > > > >>>>>>>> there
> > > > > > >>>>>>>>>> are
> > > > > > >>>>>>>>>>>> non-vector use cases that would benefit
> > > > > > >>>>>>>>>>>> from such guarantees at the storage layer.
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>> [1] https://github.com/lance-format/lance/pull/3648
> > > > > > >>>>>>>>>>>> [2]
> https://github.com/lance-format/lance/issues/8607
> > > > > > >>> (see
> > > > > > >>>>>>> point 2
> > > > > > >>>>>>>>> in
> > > > > > >>>>>>>>>>>> "root
> > > > > > >>>>>>>>>>>> cause")
> > > > > > >>>>>>>>>>>> [3]
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>
> > > > > > >>>>>>>
> > > > > > >>>>>>
> > > > > > >>>>>
> > > > > > >>>>
> > > > > > >>>
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/lancedb/lancedb/blob/0327618a62483feb6dca2336549845e5887cd48f/python/python/lancedb/table.py#L4807-L4911
> > > > > > >>>>>>>>>>>> [4]
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>
> > > > > > >>>>>>>
> > > > > > >>>>>>
> > > > > > >>>>>
> > > > > > >>>>
> > > > > > >>>
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/lancedb/lancedb/blob/0327618a62483feb6dca2336549845e5887cd48f/rust/lancedb/src/table/add_data.rs#L105-L110
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>> On Mon, Sep 28, 2026 at 6:19 PM Philipp Fischbeck
> via
> > > > > > >>> dev <
> > > > > > >>>>>>>>>>>> dev@parquet.apache.org> wrote:
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>> I also favor moving forward with the proposal as
> is.
> > > > > > >> I
> > > > > > >>>> see
> > > > > > >>>>>>> strong
> > > > > > >>>>>>>>>> value
> > > > > > >>>>>>>>>>>> and
> > > > > > >>>>>>>>>>>>> use cases for the numeric vector type, and I agree
> > > > > > >> with
> > > > > > >>>> the
> > > > > > >>>>>>>>>>> restrictions
> > > > > > >>>>>>>>>>>>> around NaN/Inf values and element types in general.
> > > > > > >>>>> Precedent
> > > > > > >>>>>>>>> exists
> > > > > > >>>>>>>>>> in
> > > > > > >>>>>>>>>>>>> other types that restrict which physical type
> values
> > > > > > >>> are
> > > > > > >>>>>>>> logically
> > > > > > >>>>>>>>>>> valid
> > > > > > >>>>>>>>>>>>> (e.g., the geospatial types needing to be
> > > > > > >> WKB-encoded).
> > > > > > >>>> A
> > > > > > >>>>>>>> general
> > > > > > >>>>>>>>>>>>> fixed-size list also has value, but I think it
> makes
> > > > > > >>>> sense
> > > > > > >>>>> to
> > > > > > >>>>>>>>>> separate
> > > > > > >>>>>>>>>>>> the
> > > > > > >>>>>>>>>>>>> two types. The semantics are relevant, and I can
> see
> > > > > > >>> how
> > > > > > >>>>> the
> > > > > > >>>>>>> two
> > > > > > >>>>>>>>>> types
> > > > > > >>>>>>>>>>>>> might diverge further in the future regarding
> > > > > > >>> statistics
> > > > > > >>>> or
> > > > > > >>>>>>>>>> encodings.
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>> I would be okay with renaming the vector type, but
> it
> > > > > > >>>> seems
> > > > > > >>>>>>> like
> > > > > > >>>>>>>>> the
> > > > > > >>>>>>>>>>>> VECTOR
> > > > > > >>>>>>>>>>>>> name is already used across other engines mentioned
> > > > > > >> in
> > > > > > >>>> this
> > > > > > >>>>>>>> thread.
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>> Best,
> > > > > > >>>>>>>>>>>>> Philipp
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>> On Sat, Sep 26, 2026 at 4:37 PM Gang Wu <
> > > > > > >>>> ustcwg@gmail.com>
> > > > > > >>>>>>>> wrote:
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>> To separate orthogonal concerns, I feel we should
> > > > > > >>> treat
> > > > > > >>>>> the
> > > > > > >>>>>>>>> general
> > > > > > >>>>>>>>>>>>>> FixedSizeList as the underlying physical layout,
> > > > > > >> and
> > > > > > >>>>>> separate
> > > > > > >>>>>>>> it
> > > > > > >>>>>>>>>> from
> > > > > > >>>>>>>>>>>> the
> > > > > > >>>>>>>>>>>>>> discussion of a specialized VECTOR (or broader ML
> > > > > > >>>> tensor)
> > > > > > >>>>>>>> logical
> > > > > > >>>>>>>>>>> type.
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>> Currently, the immediate need is to define a
> > > > > > >>>> specialized
> > > > > > >>>>>>> VECTOR
> > > > > > >>>>>>>>>> type
> > > > > > >>>>>>>>>>> so
> > > > > > >>>>>>>>>>>>>> that different engines and formats have a common
> > > > > > >>>> semantic
> > > > > > >>>>>>>>> contract
> > > > > > >>>>>>>>>>> for
> > > > > > >>>>>>>>>>>>>> interoperability.
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>> If the main point of contention is just the
> > > > > > >>>> NaN/Inf/null
> > > > > > >>>>>>>>>> limitations,
> > > > > > >>>>>>>>>>>>> maybe
> > > > > > >>>>>>>>>>>>>> we can resolve this by adding a qualification to
> > > > > > >> the
> > > > > > >>>> name
> > > > > > >>>>>>> (like
> > > > > > >>>>>>>>>>> Andrew
> > > > > > >>>>>>>>>>>>>> suggested, e.g., VECTOR_EMBEDDING or
> FINITE_VECTOR)
> > > > > > >>> to
> > > > > > >>>>>> avoid
> > > > > > >>>>>>>>>>> confusion
> > > > > > >>>>>>>>>>>>> with
> > > > > > >>>>>>>>>>>>>> broader ML use cases. Alternatively, we could
> > > > > > >>>> introduce a
> > > > > > >>>>>>>> boolean
> > > > > > >>>>>>>>>>> flag
> > > > > > >>>>>>>>>>>>>> (e.g., optional bool VectorType.finite) as
> proposed
> > > > > > >>>>>> earlier.
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>> On Fri, Sep 25, 2026 at 3:52 AM Andrew Lamb <
> > > > > > >>>>>>>>>> andrewlamb11@gmail.com>
> > > > > > >>>>>>>>>>>>>> wrote:
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>> I am a little confused about the points of
> > > > > > >>>> disagreement
> > > > > > >>>>>>> here.
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>> Is it that the name `VECTOR` is too general for
> > > > > > >> the
> > > > > > >>>>>>> specific
> > > > > > >>>>>>>>> use
> > > > > > >>>>>>>>>>> case
> > > > > > >>>>>>>>>>>>> of
> > > > > > >>>>>>>>>>>>>>> fixed length arrays with non-finite values used
> > > > > > >> for
> > > > > > >>>>>> vector
> > > > > > >>>>>>>>>>>> embeddings?
> > > > > > >>>>>>>>>>>>>> If
> > > > > > >>>>>>>>>>>>>>> so, maybe we could add a qualification to the
> > > > > > >> name
> > > > > > >>>>>>>>>> (FINITE_VECTOR?
> > > > > > >>>>>>>>>>>>>>> FEATURE_VECTOR? VECTOR_EMBEDDING?)
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>> It seems like there is precedent for defining
> > > > > > >>> logical
> > > > > > >>>>>> types
> > > > > > >>>>>>>>> that
> > > > > > >>>>>>>>>>>>> specify
> > > > > > >>>>>>>>>>>>>>> certain physical types and restrict their
> > > > > > >> contents
> > > > > > >>> to
> > > > > > >>>>>> help
> > > > > > >>>>>>>>>> readers
> > > > > > >>>>>>>>>>>>>>> interpret meaning, for example string[1] or JSON;
> > > > > > >>>>>> Defining
> > > > > > >>>>>>>> one
> > > > > > >>>>>>>>>> for
> > > > > > >>>>>>>>>>>>>> vector
> > > > > > >>>>>>>>>>>>>>> embeddings seems reasonable enough to me.
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>> Andrew
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>> [1]:
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>
> > > > > > >>>>>>>
> > > > > > >>>>>>
> > > > > > >>>>>
> > > > > > >>>>
> > > > > > >>>
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#string
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>> On Thu, Sep 24, 2026 at 3:34 PM Daniel Weeks <
> > > > > > >>>>>>>>> dweeks@apache.org>
> > > > > > >>>>>>>>>>>>> wrote:
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>> Well, Parquet is not a single-purpose format.
> > > > > > >>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>> Vector stores can choose whatever policy fits
> > > > > > >>>> their
> > > > > > >>>>>>>> usage,
> > > > > > >>>>>>>>>> but
> > > > > > >>>>>>>>>>>>> we're
> > > > > > >>>>>>>>>>>>>>>>> talking about a general-purpose file format
> > > > > > >>> that
> > > > > > >>>>> aims
> > > > > > >>>>>>> to
> > > > > > >>>>>>>> be
> > > > > > >>>>>>>>>>>> broadly
> > > > > > >>>>>>>>>>>>>>>>> applicable.
> > > > > > >>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>> I agree that the format is intended to be
> > > > > > >> general
> > > > > > >>>>>>> purpose,
> > > > > > >>>>>>>>> but
> > > > > > >>>>>>>>>>>> types
> > > > > > >>>>>>>>>>>>>>> within
> > > > > > >>>>>>>>>>>>>>>> the format define specific representation and
> > > > > > >>>>> semantics
> > > > > > >>>>>>> to
> > > > > > >>>>>>>>>>> achieve
> > > > > > >>>>>>>>>>>>> that
> > > > > > >>>>>>>>>>>>>>>> goal.
> > > > > > >>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>> "Vector" has a more general sense than ML and
> > > > > > >>>> it's
> > > > > > >>>>>>> quite
> > > > > > >>>>>>>>>>> trivial
> > > > > > >>>>>>>>>>>> to
> > > > > > >>>>>>>>>>>>>>>>> produce vectors with non-finite values using
> > > > > > >>> e.g.
> > > > > > >>>>>> NumPy
> > > > > > >>>>>>>> or
> > > > > > >>>>>>>>>>> Pandas
> > > > > > >>>>>>>>>>>>> (or
> > > > > > >>>>>>>>>>>>>>>>> any other piece of dataframe or scientific
> > > > > > >>>>> computing
> > > > > > >>>>>>>>>> software).
> > > > > > >>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>> The justification for restricting non-finite
> > > > > > >>>> values
> > > > > > >>>>> is
> > > > > > >>>>>>>>>> directly
> > > > > > >>>>>>>>>>>>>> related
> > > > > > >>>>>>>>>>>>>>> to
> > > > > > >>>>>>>>>>>>>>>> the semantic meaning of a vector.  Vectors have
> > > > > > >>>>>> specific
> > > > > > >>>>>>>>>>>>>> characteristics
> > > > > > >>>>>>>>>>>>>>>> (dimension, magnitude, direction, etc).
> > > > > > >>> Non-finite
> > > > > > >>>>>>> values
> > > > > > >>>>>>>>> are
> > > > > > >>>>>>>>>>>>> invalid
> > > > > > >>>>>>>>>>>>>>> and
> > > > > > >>>>>>>>>>>>>>>> break those characteristics.  I'd liken it to
> > > > > > >>>> storing
> > > > > > >>>>>>>>> non-utf8
> > > > > > >>>>>>>>>>>> values
> > > > > > >>>>>>>>>>>>>> in
> > > > > > >>>>>>>>>>>>>>> a
> > > > > > >>>>>>>>>>>>>>>> string and still calling it a string.
> > > > > > >>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>> That also differentiates a vector from a fixed
> > > > > > >>>> length
> > > > > > >>>>>>> list,
> > > > > > >>>>>>>>>> which
> > > > > > >>>>>>>>>>>> is
> > > > > > >>>>>>>>>>>>>> more
> > > > > > >>>>>>>>>>>>>>>> general and does not carry the same semantic
> > > > > > >>>> meaning.
> > > > > > >>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>> I also don't think we need to bundle all
> > > > > > >>> similarly
> > > > > > >>>>>> shaped
> > > > > > >>>>>>>> use
> > > > > > >>>>>>>>>>> cases
> > > > > > >>>>>>>>>>>>>> into
> > > > > > >>>>>>>>>>>>>>>> the same solution (e.g. flxed size list).
> > > > > > >> Vector
> > > > > > >>>> is
> > > > > > >>>>> a
> > > > > > >>>>>>> well
> > > > > > >>>>>>>>>>>>> established
> > > > > > >>>>>>>>>>>>>>>> data type at this point and aligns with the
> > > > > > >>> general
> > > > > > >>>>>>> purpose
> > > > > > >>>>>>>>>>> nature
> > > > > > >>>>>>>>>>>> of
> > > > > > >>>>>>>>>>>>>> the
> > > > > > >>>>>>>>>>>>>>>> format (many other formats and databases
> > > > > > >> natively
> > > > > > >>>>>> support
> > > > > > >>>>>>>>>> this).
> > > > > > >>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>> -Dan
> > > > > > >>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>> On Thu, Sep 24, 2026 at 11:55 AM Antoine
> > > > > > >> Pitrou <
> > > > > > >>>>>>>>>>>> antoine@python.org>
> > > > > > >>>>>>>>>>>>>>>> wrote:
> > > > > > >>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>> Le 24/09/2026 à 20:11, Russell Spitzer a
> > > > > > >> écrit
> > > > > > >>> :
> > > > > > >>>>>>>>>>>>>>>>>> I think something I haven't seen yet is the
> > > > > > >>>> clear
> > > > > > >>>>>>>>> use-case
> > > > > > >>>>>>>>>>> for
> > > > > > >>>>>>>>>>>>>>>> non-finite
> > > > > > >>>>>>>>>>>>>>>>>> values within a vector type. There has
> > > > > > >> been a
> > > > > > >>>> lot
> > > > > > >>>>>> of
> > > > > > >>>>>>>>>>>> theoretical
> > > > > > >>>>>>>>>>>>>>>>> discussion
> > > > > > >>>>>>>>>>>>>>>>>> here about use cases but what toolings
> > > > > > >>>> currently
> > > > > > >>>>>>>> produce
> > > > > > >>>>>>>>>>> these
> > > > > > >>>>>>>>>>>>>>> vectors
> > > > > > >>>>>>>>>>>>>>>>> and
> > > > > > >>>>>>>>>>>>>>>>>> how do they function in downstream engines?
> > > > > > >>> I'd
> > > > > > >>>>>> like
> > > > > > >>>>>>> to
> > > > > > >>>>>>>>>> limit
> > > > > > >>>>>>>>>>>>>>>> introducing
> > > > > > >>>>>>>>>>>>>>>>>> types that have a wide range of values
> > > > > > >> which
> > > > > > >>>> are
> > > > > > >>>>>>> simply
> > > > > > >>>>>>>>> are
> > > > > > >>>>>>>>>>>>> failure
> > > > > > >>>>>>>>>>>>>>>> modes
> > > > > > >>>>>>>>>>>>>>>>>> of different flavors
> > > > > > >>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>> "Vector" has a more general sense than ML and
> > > > > > >>>> it's
> > > > > > >>>>>>> quite
> > > > > > >>>>>>>>>>> trivial
> > > > > > >>>>>>>>>>>> to
> > > > > > >>>>>>>>>>>>>>>>> produce vectors with non-finite values using
> > > > > > >>> e.g.
> > > > > > >>>>>> NumPy
> > > > > > >>>>>>>> or
> > > > > > >>>>>>>>>>> Pandas
> > > > > > >>>>>>>>>>>>> (or
> > > > > > >>>>>>>>>>>>>>>>> any other piece of dataframe or scientific
> > > > > > >>>>> computing
> > > > > > >>>>>>>>>> software).
> > > > > > >>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>> It's not like the general concept of a vector
> > > > > > >>> is
> > > > > > >>>> a
> > > > > > >>>>>> IETF
> > > > > > >>>>>>>>>>> standard
> > > > > > >>>>>>>>>>>> or
> > > > > > >>>>>>>>>>>>>>>>> anything, so analogies with JSON or UTF8
> > > > > > >> (where
> > > > > > >>>> the
> > > > > > >>>>>>>> values
> > > > > > >>>>>>>>>> are
> > > > > > >>>>>>>>>>>>>> expected
> > > > > > >>>>>>>>>>>>>>>>> to be valid according to the respective
> > > > > > >> specs)
> > > > > > >>> do
> > > > > > >>>>> not
> > > > > > >>>>>>>>> apply.
> > > > > > >>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>> This discussion would IMHO have a better
> > > > > > >> chance
> > > > > > >>>> of
> > > > > > >>>>>>>>> converging
> > > > > > >>>>>>>>>>> if
> > > > > > >>>>>>>>>>>> we
> > > > > > >>>>>>>>>>>>>>>>> separated the two orthogonal concerns of:
> > > > > > >>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>> 1. representing fixed-length list/vector data
> > > > > > >>> in
> > > > > > >>>>>>> Parquet
> > > > > > >>>>>>>>>> (with
> > > > > > >>>>>>>>>>> a
> > > > > > >>>>>>>>>>>>>>>>> potential limitation to non-nested types)
> > > > > > >>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>> 2. representing the information that a given
> > > > > > >>>> column
> > > > > > >>>>>> (or
> > > > > > >>>>>>>>>> column
> > > > > > >>>>>>>>>>>>> chunk,
> > > > > > >>>>>>>>>>>>>>>>> etc.) does not contain non-finite values
> > > > > > >>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>> ... both of which are useful on their own
> > > > > > >>> right,
> > > > > > >>>>> but
> > > > > > >>>>>> do
> > > > > > >>>>>>>> not
> > > > > > >>>>>>>>>>> have
> > > > > > >>>>>>>>>>>> to
> > > > > > >>>>>>>>>>>>>> be
> > > > > > >>>>>>>>>>>>>>>>> conflated and bundled together.
> > > > > > >>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>> Regards
> > > > > > >>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>> Antoine.
> > > > > > >>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>
> > > > > > >>>>>>>
> > > > > > >>>>>>
> > > > > > >>>>>
> > > > > > >>>>
> > > > > > >>>
> > > > > > >>
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] Numeric Vector type: physical representation (outcome of focused call)

[Russell Spitzer <ru...@gmail.com>]

I support proceeding with the two types. I honestly think the cost of
adding new logical types is overestimated. If we have as many interested
parties as have appeared in this thread we already know there is a large
group of engineers ready to do the work and engines ready to consume (and
produce) the type. The fact that implementations currently use heuristics,
sampling (I loved Gunnar's talk on his hardwood work on this at CoC
Glasgow) or custom annotations to achieve the same performance benefits
makes me think we should just create the type and move on.

On Mon, Oct 19, 2026 at 7:21 PM Ryan Blue <bl...@apache.org> wrote:

> An annotation on list<float> is useful because:
> - We want to correctly annotate what is being stored
> - We want to unlock optimizations when each non-null list has the same
> number of elements
> - We want to have writer guarantees
>
> I don't think that logical type compatibility is a good argument. We want
> to avoid a case where we have to treat all binary like a string. And
> because we are introducing this in table formats after standardizing it
> here, we can do that reliably.
>
> As I said in my last email, I think there's a difference between finiteness
> and general "invariants" you might want. Vectors aren't vectors if they
> don't have a length and a direction. Other invariants don't work the same
> way. A zero-length vector is still a valid vector and can be used, just not
> for certain distance metrics.
>
> I also want to clarify that when we looked into this more closely and
> recognized the mismatch between the two separate use cases, I offered to
> split out the finite vector work so that we didn't cause any headaches for
> people who are interested in fixed-length list. It sounds like not
> splitting out the use cases is causing a lot of headache. Vectors for our
> use case need a finiteness guarantee at write time, and fixed-length lists
> need to store NaN and other values.
>
> It's okay to have two types and we should for two different use cases.
> Otherwise, we are forcing one to make an unnecessary compromise. I don't
> want to force everyone to exclude NaN and infinite values from fixed-length
> lists. Seems reasonable that we should also be able to have vectors that
> exclude those values, since that is how they are used and we have to also
> guarantee it for exchanging data with other existing databases.
>
> Ryan
>
> On Mon, Oct 19, 2026 at 3:12 AM Will Edwards via dev <
> dev@parquet.apache.org>
> wrote:
>
> > It reads well. Three small fixes:
> >
> > 1. **"back in May or so"** is ambiguous. The fixed-size-list-for-tensors
> > framing is from Rok's opening post in May 2024. The May 2026 design doc
> was
> > already mostly about performance and layout. "Originally" avoids anyone
> > correcting the date.
> > 2. **"Spark's sparse feature vectors routinely use NaN"** isn't quite
> > right. The NaN-for-missing behaviour comes from `VectorAssembler` and
> > `Imputer` and applies to Spark ML vectors generally, dense or sparse.
> > "Spark's feature vectors, dense and sparse," fixes it.
> > 3. There are typos: "Iirc" should be "IIRC", and "Spark, BigQuery and
> > DuckDB and pretty much…" has a doubled "and".
> >
> > The last sentence of the invariants paragraph could be clearer. "Although
> > that doesn't mean readers shouldn't check on read" has a double negative
> > and is easy to misread. "Though careful readers will still check on read"
> > is simpler.
> >
> > With those applied:
> >
> > ---
> >
> > Hi all,
> >
> > The scope of this discussion keeps moving :). IIRC it originally was
> about
> > fixed-size lists for ML tensors, became a performance question, and is
> now
> > about whether vectors are mathematically valid. Each step made sense,
> but I
> > don't think we have consensus on where we've ended up?
> >
> > I think there is much useful work in what we left behind. Many ML vectors
> > aren't embeddings for search. E.g. Spark's feature vectors, dense and
> > sparse, routinely use NaN to mark missing values, and quantised formats
> > pack codes with scales in ways Parquet can't describe today. ML-oriented
> > formats like Lance are moving towards tensors, quantisation and indexes.
> A
> > type that told engines how to interpret that kind of data would enable
> new
> > things. An annotation on LIST<FLOAT> doesn't.
> >
> > My big concern with the current focus on embeddings needing a logical
> type
> > is compatibility. Today LIST<FLOAT> is the common format for embeddings,
> > and Spark, BigQuery, DuckDB and pretty much every other library I can
> think
> > of offer distance functions on plain lists, so any engine can use any
> file.
> > If engines start requiring a VECTOR annotation, existing files and
> anything
> > rewritten by a tool that doesn't know the type stop working. If they
> don't
> > require it, the annotation adds nothing. We've seen this before with
> > strings written without the UTF8 annotation, which engines still have to
> > cope with years later.
> >
> > On invariants, I agree some users may want the writer to check what they
> > write as they write it, but that can be a writer option rather than a
> type.
> > Readers that want evidence could get it from extra statistics or
> metadata,
> > though careful readers will still check on read.
> >
> > If a finite fixed-length vector contract is useful to table formats,
> > perhaps Iceberg is the place for it, with Parquet storing plain lists.
> I'd
> > raise the same worry there, though: if tables start rejecting untagged
> > columns, we've just moved the split up a layer.
> >
> > Best,
> > Will
> >
> > On Sat, 17 Oct 2026 at 00:10, Ryan Blue <bl...@apache.org> wrote:
> >
> > > I agree with much of what Dan said, but I'll add some to this thread as
> > > well.
> > >
> > > First, I want to address the arguments that were pointed out:
> > > 1. Other vector databases have finiteness requirements
> > > 2. "Vectors are the hot stuff right now" -- let's jump on the bandwagon
> > > 3. A general solution will take too long, finiteness will make faster
> > > progress
> > > 4. To give a guarantee to readers that no INF/NaN/NULL will be in the
> > > vector
> > >
> > > I don't think these represent our actual motivation or position, which
> > > makes them function as a straw-man. This is probably not what was
> > > intended, but I want to make sure that the misunderstanding
> > > doesn't continue. For one thing, we aren't saying that we need finite
> > > vectors because they are "hot stuff".
> > >
> > > We want to build tools that use them on top of Parquet. I think it
> should
> > > also be clear that we aren't advocating for finite elements blindly
> just
> > > because other engines do (cargo cult design!). There are real
> > > interoperability issues if we choose to go a different direction so
> this
> > is
> > > a "Chesterton's Fence" situation. It's important to understand why
> > > those databases
> > > made that choice. I've also pointed this out as confirmation that we're
> > > going down a reasonable and established path.
> > >
> > > The third argument sounds like a simple misunderstanding. We're
> pursuing
> > a
> > > logical type right now because that's forward compatible and we want to
> > > label the data properly and set expectations for the type. Physical
> type
> > > changes break forward compatibility and need to be released in a new
> > format
> > > version . I think this is the source of the confusion since it has been
> > in
> > > the same discussion, but the choice about finiteness is separate.
> > >
> > > That brings us to the last argument, whether there is value in writers
> > > participating in guarantees. I think there's value in adding
> requirements
> > > for writers. It's up to readers how they want to use those
> requirements,
> > > but a writer requirement allows a policy of "garbage in garbage out"
> > rather
> > > than being forced to handle bad data. Why would a reader need to run
> > > validation on every element just in case a writer had a bug? And why
> > should
> > > the potential for writer bugs mean that it has to be the reader's
> > problem?
> > > If a writer produces bad data, we can absolutely reject it and tell the
> > > writer to fix the bug.
> > >
> > > I'm also puzzled by the assertion that writer guarantees have value,
> but
> > > only if there are other, more specific guarantees in addition. This
> seems
> > > to cede the point that a contract between readers and writers makes
> > sense,
> > > but only if it is a _specific_ contract. But the proposed additions
> don't
> > > make sense in the same way.
> > >
> > > Take, for example, the suggestion to also disallow zero-length vectors.
> > At
> > > a glance this seems similar, but I think it assumes a specific context
> > and
> > > is better suited to be a constraint for some use cases rather than part
> > of
> > > what is considered valid for the type. That's because zero-length
> vectors
> > > are valid vectors with a length. You can calculate Euclidean or
> Manhattan
> > > distance and use them in many contexts. They don't work if you're
> > > normalizing to a length of 1, but it's a choice to normalize and use a
> > dot
> > > product to calculate cosine distance. And at this point, we're talking
> > > about the usefulness of restrictions for the type, not a Platonic
> ideal.
> > >
> > > I think the restriction that makes the most sense is that vectors have
> a
> > > length and can be compared in distance calculations. 0-length vectors
> are
> > > valid because they point somewhere and have a length. We shouldn't
> > restrict
> > > uses to only cosine distance over normalized vectors. Those are use
> case
> > > dependent choices and there are other valid ones. I also don't think
> that
> > > the model used to produce a vector is part of the vector type, just
> like
> > > the method used to produce a metric doesn't define its type: cosine
> > > similarity produces values between -1 and 1, but we store it in a
> > `float`,
> > > not a `float(cosine)`.
> > >
> > > Why does it make sense to have a finiteness restriction as part of the
> > > type? Because that's what a (mathematical) vector is. A simple search
> > gives
> > > a pretty common definition: "A vector is an object that has both
> > magnitude
> > > and a direction". If a dimension/element is NaN, null, or infinite, the
> > > vector doesn't point anywhere and it doesn't have a magnitude. Not only
> > can
> > > it not be used, it doesn't meet the most basic definition of the type.
> > >
> > > Ryan
> > >
> > > On Thu, Oct 15, 2026 at 1:07 AM Jan Finis <jp...@gmail.com> wrote:
> > >
> > > > >
> > > > >
> > > > >
> > > > Daniel,
> > > >
> > > > alright, so no physical optimizations intended.  We only want this
> for
> > > > semantic reasons, not for any significant physical optimization. If
> > we're
> > > > all on the same page here, that is good 👍.
> > > >
> > > > To phrase my remaining problem with this more concretely:
> > > > *Let's say we add this type. What cool new stuff can we now do that
> we
> > > > couldn't do before?*
> > > > That is the litmus test that a type has to pass to be useful IMHO.
> And
> > > here
> > > > I can't come up with a tangible use case that wasn't possible before.
> > > > Especially if we one day have fixed size lists in Parquet, the
> > remaining
> > > > niche for this type becomes so small, that the added complexity isn't
> > > worth
> > > > the benefit IMHO.
> > > >
> > > > As I guess most people on here are active maintainers of a Parquet
> > > > implementation: Concretely, what would you be using this type for if
> we
> > > had
> > > > it? Would you allow math only on fields tagged with this type? What
> if
> > > that
> > > > math yields INF or NaN? Not allowed?
> > > >
> > > > Especially if we also discuss having both VECTOR and FINITE_VECTOR.
> How
> > > > will you guys wire this up with the type systems of your engines? Do
> > your
> > > > engines also have two types here? Do you type-decay a FINITE_VECTOR
> to
> > a
> > > > VECTOR once someone does a multiplication that could yield INF? By
> > > itself,
> > > > a FINITE_VECTOR sounds good. Holistically looked at from a type
> system
> > > > perspective, it comes with a lot of complexity. Do the benefits
> really
> > > > outweigh the complexity here? That's why I would rather have the bare
> > > > minimum: A fixed size array (list) of floats or at least just a
> vector
> > > > without finite. Then all these detailed questions go away.
> > > >
> > > > I'll take this one step further... Why is there any debate about this
> > at
> > > > > all? Why can't a type simply be added to the *specification* as an
> > > > > administrative detail and be done?
> > > >
> > > >
> > > > Andrew,
> > > >
> > > > Parquet is supposed to be a lingua franca of data. The specification
> is
> > > the
> > > > authoritative truth for this language and all implementations are
> > > expected
> > > > to implement as much as possible from this spec, so anything in the
> > spec
> > > > does causes implementation effort. So just adding anything to this
> spec
> > > > without debate is not a good idea. It is even the other way around:
> We
> > > need
> > > > to have way higher rigor with the spec than we need to have with any
> > > > implementations. Implementations can be changed, the spec needs to be
> > > > stable.
> > > >
> > > > A tag-only type that doesn't come with any restrictions can be added
> > with
> > > > little effort. But here we want to tie restrictions to it (no
> NaN/Inf),
> > > so
> > > > we would expect implementations to enforce these. So there is some
> > > > implementation effort expected here.
> > > >
> > > > Also, even if we only wanted a tag type, I would still argue that you
> > > > shouldn't add those without a debate. What if A wants XML, B wants
> > YAML,
> > > C
> > > > wants quaternions, D wants address, E wants phone number. If we just
> > > added
> > > > any types that are "just tags", Parquet logical types would just
> > become a
> > > > trash dump of any type that anyone ever thought to be useful. So
> > rigorous
> > > > assessment of each new type candidate is necessary.
> > > >
> > > > Finally, many engines couple parquet with another type system (e.g.,
> > > Arrow
> > > > or SQL). So even with a tag only type, you at least still have the
> > > > implementation burden of wiring this up with your engine's type
> system.
> > > >
> > > > So, just because logical types are "just" logical, that doesn't mean
> we
> > > can
> > > > add them without any rigor.
> > > >
> > > >
> > > > A few naive questions:
> > > > > 1. Would a physical definition with NANs & Inf+/- allow efficient
> > > > > encodings like ALP?
> > > > > 2. If the physical representation allows values that make encodings
> > > like
> > > > > ALP impossible, can a tighter Logical type allow different
> encodings?
> > > > > 3. Can we RLE or binpack a vector?
> > > >
> > > >
> > > > The simple general answer here is that all these things do not
> require
> > a
> > > > logical type. Writers can already do any such optimization if the
> > actual
> > > > data contains no NaNs & Infs. All these physical optimizations just
> > > require
> > > > no actual NaNs/Infs to be present in a page, they don't need a
> > guarantee
> > > > that the whole file won't contain any.
> > > >
> > > > Cheers,
> > > > Jan
> > > >
> > > >
> > > >
> > > >
> > > > Am Do., 15. Okt. 2026 um 04:28 Uhr schrieb <kurtiscwright@gmail.com
> >:
> > > >
> > > > > Hi Everyone,
> > > > >
> > > > > Glad to see a Vector type being added. I have a few use cases I
> want
> > to
> > > > > pursue once we have an agreed upon definition.
> > > > >
> > > > > In regards to that definition discussion. I feel comfortable with
> the
> > > > idea
> > > > > of the semantic meaning Dan raises though I can’t shake a slight
> > > > discomfort
> > > > > (maybe FOMO) with limiting values in the physical definition.
> > > > >
> > > > > A few naive questions:
> > > > > 1. Would a physical definition with NANs & Inf+/- allow efficient
> > > > > encodings like ALP?
> > > > > 2. If the physical representation allows values that make encodings
> > > like
> > > > > ALP impossible, can a tighter Logical type allow different
> encodings?
> > > > > 3. Can we RLE or binpack a vector?
> > > > >
> > > > > Best Regards,
> > > > > Kurtis C. Wright
> > > > > > On Oct 14, 2026, at 12:45, Daniel Weeks <dw...@apache.org>
> wrote:
> > > > > >
> > > > > > Hey Jan,
> > > > > >
> > > > > > There's a lot that I disagree with respect to the framing of
> these
> > > > > > arguments and how we should be thinking about introducing vector
> > as a
> > > > > > logical type.  There's a repeated tendency to let the physical
> > > > > > representation or performance drive discussion around types as
> > > opposed
> > > > to
> > > > > > what it represents.
> > > > > >
> > > > > > In parquet, establishing semantic meaning is the purpose of
> logical
> > > > > types.
> > > > > > The logical type separates how to interpret a value from its
> > physical
> > > > > > representation.  We can represent a timestamp with INT64 or even
> a
> > > > string
> > > > > > value, so why have a type for it? Even though a FIXED_SIZED_LIST
> > and
> > > > > VECTOR
> > > > > > may have the exact same representation and performance, they have
> > > > > different
> > > > > > semantic meaning.  Databases and engines across the ecosystem
> > already
> > > > > > recognize this differentiation and have separate representations.
> > > The
> > > > > > choice to separate these isn't because the discussion is too
> > tedious;
> > > > > > they're fundamentally different things.
> > > > > >
> > > > > > The finite requirement is what makes a vector a reflection of
> what
> > it
> > > > > > represents.  As we've discussed in the thread, non-finite values
> in
> > > the
> > > > > > representation violate the properties of a vector and the value
> > > ceases
> > > > to
> > > > > > be a vector.  On the other hand, a 0 vector is still a vector
> (you
> > > can
> > > > > > argue the utility of it, but it is a valid vector).
> > Implementations
> > > > can
> > > > > > choose what they validate or how to handle execution errors;
> that's
> > > an
> > > > > > implementation concern, not a requirement.  Operations performed
> > on a
> > > > > > vector depend on the context in which they are used, but the
> > > > > representation
> > > > > > doesn't need to solve scenarios where that application doesn't
> make
> > > > > sense.
> > > > > >
> > > > > > The guarantees or context needed to use a vector do not need to
> be
> > > > > inherent
> > > > > > in the type for it to be a useful type.  We don't need the
> currency
> > > > used
> > > > > > for decimal or whether a floating point is a normalized value as
> > part
> > > > of
> > > > > > the type to make use of those types.  This conflates the use
> cases
> > > with
> > > > > the
> > > > > > type system and unnecessarily attempts to draw the context into
> the
> > > > type
> > > > > > definition.  Similarly, you don't need to capture normalization,
> > > model,
> > > > > > etc. as part of the type for it to be useful.
> > > > > >
> > > > > > So the reason we want to add a vector type (or FINITE_VECTOR) is
> > > > because
> > > > > we
> > > > > > want to store vectors with the semantic meaning and requirements
> > > based
> > > > on
> > > > > > what they represent.
> > > > > >
> > > > > > -Dan
> > > > > >
> > > > > >
> > > > > >> On Wed, Oct 14, 2026 at 1:22 AM Jan Finis <jp...@gmail.com>
> > > wrote:
> > > > > >>
> > > > > >> A bit late to the party, but here we go:
> > > > > >>
> > > > > >> I have the strong sentiment that the FINITE_VECTOR type as it is
> > > > > currently
> > > > > >> being discussed will not be very useful and just increase
> > > > implementation
> > > > > >> complexity. The arguments for the type are all quite weak
> (details
> > > see
> > > > > >> below). The FINITE part doesn't seem to yield any value and is
> > not a
> > > > > useful
> > > > > >> restriction by itself. The vector (i.e. fixed length) part
> should
> > > > > rather be
> > > > > >> generalized instead of making a one-off thing. In general, the
> > > > existing
> > > > > >> types are already working well for uncompressed vectors. The
> real
> > > > > challenge
> > > > > >> that would bring real benefits is supporting quantization. This
> is
> > > > where
> > > > > >> you get orders of magnitude improvements to storage space and
> > > > processing
> > > > > >> speed.
> > > > > >>
> > > > > >> Let's first discuss the FINITE part. So far, the only arguments
> I
> > > have
> > > > > read
> > > > > >> for this are:
> > > > > >>
> > > > > >> 1. many other vector databases have it
> > > > > >> 2. vectors are the hot stuff right now, so Parquet better boast
> > > having
> > > > > >> "native" type support for vectors
> > > > > >> 3. by focussing on this common special case, we can make rapid
> > > > progress,
> > > > > >> while a general solution would take too long to specify
> > > > > >> 4. to give a guarantee to readers that no INF/NaN/NULL will be
> in
> > > the
> > > > > >> vector
> > > > > >>
> > > > > >> the first two arguments are quite weak by themselves. Vector
> > > databases
> > > > > are
> > > > > >> very specialized engines. Such engines can of course be very
> > > > > opinionated on
> > > > > >> their data types. A general format like Parquet must instead
> weigh
> > > > > whether
> > > > > >> a special type is worth the added complexity over a more
> general,
> > > > widely
> > > > > >> applicable type. In this case, it is IMHO not at all. And just
> > > having
> > > > a
> > > > > >> type called "vector" for advertisement reasons to make Parquet
> > > appear
> > > > > more
> > > > > >> modern seems a non-goal in the first place.
> > > > > >>
> > > > > >> The third "time to market" argument is valid for systems that
> > don't
> > > > > need a
> > > > > >> stable interface. It is a very dangerous attitude for formats
> like
> > > > > Parquet
> > > > > >> that must be stable long term and should have a consolidated,
> > > concise,
> > > > > >> non-redundant type system. For such long term storage formats,
> we
> > > need
> > > > > to
> > > > > >> take the time to build the right thing, as any shortcuts we take
> > > will
> > > > > stay
> > > > > >> as blemishes in the format forever. If I read the discussion
> > > > correctly,
> > > > > >> this is actually the main argument as of now: We have found that
> > it
> > > is
> > > > > hard
> > > > > >> to align on how a fixed size list type should look like, so we
> > want
> > > to
> > > > > go
> > > > > >> the easier route of a specialized type. This seems like a
> > > > > >> dangerous shortcut fueled by feature recency bias. Ask yourself:
> > > Would
> > > > > we
> > > > > >> want this, if Parquet already had a fixed size list type?
> > > > > >>
> > > > > >> So finally, the only technical argument is the fourth one: "It
> > > signals
> > > > > >> semantics and ensures that no bad data lands in the format". I
> see
> > > two
> > > > > big
> > > > > >> problems with this:
> > > > > >>
> > > > > >> 1. Readers *still* need to test for NaN / INF because Parquet
> > files
> > > > may
> > > > > be
> > > > > >> corrupt. Nothing can stop a malicious or buggy writer from
> > > physically
> > > > > >> writing NaN or INF. *So you save no single line of code in a
> > reader.
> > > > You
> > > > > >> save no branches, as you still need to handle the INF/NaN case.*
> > So
> > > > this
> > > > > >> doesn't help with performance at all. The only difference is,
> that
> > > the
> > > > > >> error message changes from "found an invalid INF value in the
> > data"
> > > to
> > > > > >> "Your parquet file seems to be corrupt". It is actually quite
> the
> > > > > opposite,
> > > > > >> readers and writers get more complex having to handle this new
> > type.
> > > > > >>
> > > > > >> If you now argue that it's not about performance, it's about
> > defined
> > > > > >> semantics and erroring out early in writers, then the second
> > problem
> > > > > >> arises:
> > > > > >>
> > > > > >> 2. The proposed no-NaN/INF/NULL guarantee is just one of
> multiple
> > > > > >> guarantees that you need for sensible vector processing. If "no
> > > > > >> INF/NaN/NULL" was the only guarantee you needed, I would agree
> > that
> > > > > >> FINITE_VECTOR is useful, but it is by no means the only
> guarantee
> > > you
> > > > > need,
> > > > > >> instead, depending on the distance metric, you also need the
> > > following
> > > > > >> guarantees. E.g.:
> > > > > >>
> > > > > >>   - No vector may be the 0 vector, otherwise, e.g., cosine
> > > similarity
> > > > is
> > > > > >>   undefined. Cosine similarity is often used as default, so this
> > is
> > > > > >> actually
> > > > > >>   quite important in practice.
> > > > > >>   - If we want to use a dot product, then all vectors need to
> > > > guarantee
> > > > > a
> > > > > >>   length of 1. This would actually be quite useful for
> performance
> > > > > >> reasons.
> > > > > >>   - The vectors must come from the same model as the vector used
> > for
> > > > the
> > > > > >>   search. Otherwise, the search yields a numeric result, but
> that
> > > > result
> > > > > >> has
> > > > > >>   no sensible meaning.
> > > > > >>
> > > > > >> So, what use is a type that just guarantees part of the things
> you
> > > > need
> > > > > for
> > > > > >> vector search but not others? You still need additional
> guarantees
> > > in
> > > > > >> addition to that type. What makes INF/NaN so important in
> > comparison
> > > > to,
> > > > > >> say, "no 0 vector present", that it warrants an own type, while
> > the
> > > > > other
> > > > > >> guarantees do not? If we really want this type as a basis for
> > > > > cross-engine
> > > > > >> sharing of vectors, then it needs to give *all* guarantees.
> > > > > >>
> > > > > >> I'm not against a specialized vector type if it really carries
> > *all*
> > > > > >> information to be useful by itself. If it carried all that
> > > > information,
> > > > > >> then a reading engine could indeed validate that only meaningful
> > > > vector
> > > > > >> searches are executed. But as long as it is just a half-baked
> > thing,
> > > > it
> > > > > is
> > > > > >> not useful by itself and just a burden on each implementer and
> > > > > maintainer.
> > > > > >> If we need extra information anyway, why not just stick to a
> > general
> > > > > type
> > > > > >> like VECTOR (without FINITE).
> > > > > >>
> > > > > >> If you're a proponent of FINITE_VECTOR, ask yourself (or tell
> us)
> > > what
> > > > > >> advantage you will *really* get over just a (potentially fixed
> > size)
> > > > > List
> > > > > >> of Float? As discussed, it's not performance. It's also not a
> > > semantic
> > > > > >> marker that is sufficient by itself. It's just a small hint that
> > > > allows
> > > > > >> writers to error out once they see a NaN or INF in the input,
> > while
> > > > they
> > > > > >> will still happily write a bogus 0 vector. This seems to be of
> > zero
> > > > > >> benefit.
> > > > > >>
> > > > > >> Anecdotally, we currently handle petabytes of vectors in
> > production
> > > in
> > > > > >> Data360. We simply store them in a List of Float, with an
> > optimized
> > > > > R-Level
> > > > > >> reader implementation that rapidly checks that all vectors are
> the
> > > > same
> > > > > >> length. This solution works just fine. The overhead of R-level
> > > > decoding
> > > > > is
> > > > > >> barely measurable. The check is super fast and R-levels are tiny
> > in
> > > > > >> comparison to the payload vectors, so also the added I/O is
> > > > negligible.
> > > > > >> This is also why I'm so firm in my sentiment: We are running a
> > > > > production
> > > > > >> grade system processing trillions of vectors right now and see
> no
> > > big
> > > > > >> issues with the types that Parquet currently provides.
> > > > > >>
> > > > > >> As I don't just want to be a naysayer and point out problems
> with
> > > the
> > > > > >> current design, here are the directions that I would propose
> > indeed:
> > > > > >>
> > > > > >>   - a general fixed size list type. Don't do shortcuts just
> > because
> > > > the
> > > > > >>   discussion is too tedious.
> > > > > >>   - a vector type that indeed carries enough information to be
> > > really
> > > > > >>   useful for vector search by itself. (e.g., model name /
> version,
> > > > > >> optional
> > > > > >>   guarantees for no 0 vector / normalized vector length). With
> > such
> > > a
> > > > > >> type,
> > > > > >>   you could really do well defined vector searches even if the
> > > writing
> > > > > >> engine
> > > > > >>   is completely different to the reading one.
> > > > > >>   - Standardize quantizations. This will be a real game changer
> > for
> > > > > >>   performance.
> > > > > >>
> > > > > >> Cheers,
> > > > > >> Jan
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >>> Am Mi., 14. Okt. 2026 um 03:36 Uhr schrieb Gang Wu <
> > > ustcwg@gmail.com
> > > > >:
> > > > > >>>
> > > > > >>> I think separating them into FINITE_VECTOR and
> FIXED_LENGTH_LIST
> > > is a
> > > > > >> good
> > > > > >>> idea, though I think renaming FINITE_VECTOR to VECTOR looks
> more
> > > > > natural.
> > > > > >>> Should we move forward with proposing FINITE_VECTOR/VECTOR
> first?
> > > > > >>>
> > > > > >>> On Tue, Oct 13, 2026 at 6:26 PM Alkis Evlogimenos via dev <
> > > > > >>> dev@parquet.apache.org> wrote:
> > > > > >>>
> > > > > >>>> I suggest we go with VECTOR for the finite vector to keep the
> > name
> > > > the
> > > > > >>> same
> > > > > >>>> as other engines/type systems. If we want a non-finite one in
> > the
> > > > > >> future
> > > > > >>> we
> > > > > >>>> can name that something else or we add fixed size list that
> will
> > > > serve
> > > > > >>> the
> > > > > >>>> same purpose without a special name.
> > > > > >>>>
> > > > > >>>>
> > > > > >>>> On Thu, Oct 8, 2026 at 10:20 PM Ryan Blue <bl...@apache.org>
> > > wrote:
> > > > > >>>>
> > > > > >>>>> A logical type would:
> > > > > >>>>> - Describe data for users and readers to understand what the
> > file
> > > > > >>>> contains
> > > > > >>>>> - Signal that repetition and definition levels will reflect a
> > > fixed
> > > > > >>>> number
> > > > > >>>>> of elements, which implementations can use to optimize
> decoding
> > > > > >>>>> - Guarantee that there are no null elements (because the type
> > > > cannot
> > > > > >> be
> > > > > >>>>> applied to a list with optional elements)
> > > > > >>>>> - [FINITE_VECTOR] Guarantee that no elements are NaN or
> > Infinity
> > > > > >> values
> > > > > >>>> so
> > > > > >>>>> that the vectors have a finite length and are suitable for
> > > distance
> > > > > >>>>> calculations
> > > > > >>>>>
> > > > > >>>>> The finite value restriction is a restriction on writers to
> > > prevent
> > > > > >> bad
> > > > > >>>>> data from being written.
> > > > > >>>>>
> > > > > >>>>> Ignoring logical types that are not understood is a rule for
> > > > > >> readers. A
> > > > > >>>>> reader can handle vectors and pass it on to another layer.
> For
> > > > > >>> instance,
> > > > > >>>> a
> > > > > >>>>> Parquet reader that doesn't support geometry passes values as
> > > > > >> `binary`.
> > > > > >>>>> That binary value can be passed to a geometry UDF, even if
> the
> > > > > >>> underlying
> > > > > >>>>> engine and Parquet reader doesn't have support.
> > > > > >>>>>
> > > > > >>>>> A tool that doesn't understand FINITE_VECTOR can similarly
> > pass a
> > > > > >> list
> > > > > >>> of
> > > > > >>>>> floats along to a UDF. But it is still important to the user
> > that
> > > > bad
> > > > > >>>>> values are not present in the data. In your example, bad
> values
> > > are
> > > > > >>>> vectors
> > > > > >>>>> that cause the centroid calculation to fail, like vectors
> with
> > > > > >> infinite
> > > > > >>>>> length.
> > > > > >>>>>
> > > > > >>>>> Ryan
> > > > > >>>>>
> > > > > >>>>> On Thu, Oct 8, 2026 at 12:54 PM Will Edwards via dev <
> > > > > >>>>> dev@parquet.apache.org>
> > > > > >>>>> wrote:
> > > > > >>>>>
> > > > > >>>>>> Could we clarify the use cases for a VECTOR annotation on
> > > > > >>> LIST<FLOAT>?
> > > > > >>>>>> Specifically, what does it make possible that's impossible
> > > today,
> > > > > >> and
> > > > > >>>>> what
> > > > > >>>>>> does it make easier that's currently hard?
> > > > > >>>>>>
> > > > > >>>>>> I'm struggling to find a strong answer. I'd like every
> engine
> > to
> > > > > >>> offer
> > > > > >>>>>> vector functions on lists of floats without needing a new
> > > logical
> > > > > >>> type.
> > > > > >>>>>>
> > > > > >>>>>> To be clear, I'm still enthusiastic about a vector type that
> > > > > >>> describes
> > > > > >>>>> how
> > > > > >>>>>> to unpack and scale composite representations (quantised
> codes
> > > > plus
> > > > > >>>>> scales,
> > > > > >>>>>> codebooks, etc.), in the direction ML is going. That adds
> > > > > >> information
> > > > > >>>>>> engines don't have. A label on LIST<FLOAT> doesn't: engines
> > > > already
> > > > > >>>>>> understand the physical type, and many already offer vector
> > > > > >> functions
> > > > > >>>> on
> > > > > >>>>>> it. I don't see a performance benefit, and NaN/null/Inf
> checks
> > > are
> > > > > >>>> cheap
> > > > > >>>>>> enough that engines should do them anyway. The label also
> > > doesn't
> > > > > >> say
> > > > > >>>>> what
> > > > > >>>>>> you can do with the column. Knowing that a column holds
> > > embeddings
> > > > > >>>>> doesn't
> > > > > >>>>>> say how they can be used, just as zipcode and age are both
> > > > integers
> > > > > >>> but
> > > > > >>>>>> multiplying them is meaningless.
> > > > > >>>>>>
> > > > > >>>>>> My concern is the opposite effect. We're establishing that
> > > engines
> > > > > >>>> should
> > > > > >>>>>> ignore logical types they don't understand. So a tool that
> > > doesn't
> > > > > >>> know
> > > > > >>>>>> VECTOR will read the column as LIST<FLOAT>, let the user
> > compute
> > > > > >>>>> centroids
> > > > > >>>>>> (say), and write plain LIST<FLOAT> back out. If other
> engines
> > > > > >>> restrict
> > > > > >>>>>> embedding functions to VECTOR-annotated columns, that file
> > stops
> > > > > >>>> working
> > > > > >>>>>> with them, through no fault of the user. An annotation that
> > > > doesn't
> > > > > >>>>> change
> > > > > >>>>>> what the data is shouldn't be able to make it unusable.
> > > > > >>>>>>
> > > > > >>>>>> Am I missing a use case?
> > > > > >>>>>>
> > > > > >>>>>> Best, Will
> > > > > >>>>>>
> > > > > >>>>>> On Thu, 8 Oct 2026 at 19:53, Steven Wu <
> stevenz3wu@gmail.com>
> > > > > >> wrote:
> > > > > >>>>>>
> > > > > >>>>>>> I also think a FINITE_VECTOR type brings a lot of values to
> > > many
> > > > > >>> use
> > > > > >>>>>> cases.
> > > > > >>>>>>> It is would be great to make progress on the FINITE_VECTOR,
> > and
> > > > > >> not
> > > > > >>>> to
> > > > > >>>>>> fit
> > > > > >>>>>>> every use cases into the FIXED_LENGTH_LIST.
> > > > > >>>>>>>
> > > > > >>>>>>> On Mon, Oct 5, 2026 at 1:26 PM Ryan Blue <bl...@apache.org>
> > > > > >> wrote:
> > > > > >>>>>>>
> > > > > >>>>>>>> Going through this thread again, it looks like there are a
> > few
> > > > > >>>> things
> > > > > >>>>>>> that
> > > > > >>>>>>>> people generally agree on:
> > > > > >>>>>>>>
> > > > > >>>>>>>> - In the short term, we want to introduce a logical type
> or
> > > > > >>> types;
> > > > > >>>> a
> > > > > >>>>>>> better
> > > > > >>>>>>>> physical representation can be done later. We can focus on
> > > > > >>> logical
> > > > > >>>>>> types.
> > > > > >>>>>>>> - There is precedent in logical types for restricting
> > values,
> > > > > >>> like
> > > > > >>>>> UTF8
> > > > > >>>>>>>> strings to contain only UTF-8 codepoints. I think we also
> > > agree
> > > > > >>>> that
> > > > > >>>>> we
> > > > > >>>>>>>> could restrict vector elements to non-nested or primitive
> > > > > >> types.
> > > > > >>>>>>>> - There are many examples of databases that restrict their
> > > > > >> vector
> > > > > >>>>> types
> > > > > >>>>>>> to
> > > > > >>>>>>>> non-null, finite elements.
> > > > > >>>>>>>> - Some people want to have write-time guarantees that
> > vectors
> > > > > >>> have
> > > > > >>>>> only
> > > > > >>>>>>>> non-null, finite elements
> > > > > >>>>>>>> - Some people want to extend this work to include tensors,
> > > > > >>> shorter
> > > > > >>>>>>> floating
> > > > > >>>>>>>> point representations, variable lengths, NaN or Infinity
> > > > > >>> elements,
> > > > > >>>>> etc.
> > > > > >>>>>>>>
> > > > > >>>>>>>> It sounds like we are nearly there, but have two halves of
> > the
> > > > > >>>>>> community
> > > > > >>>>>>>> that care about different things. That just means we
> should
> > > > > >> have
> > > > > >>>>>> multiple
> > > > > >>>>>>>> types. It's entirely reasonable to want a type that allows
> > NaN
> > > > > >>>>> values,
> > > > > >>>>>>> and
> > > > > >>>>>>>> it's also reasonable for those of us building databases,
> > table
> > > > > >>>>>> standards,
> > > > > >>>>>>>> and vector indexes to care about guarantees that elements
> > are
> > > > > >>>> finite.
> > > > > >>>>>>>> There's more than enough examples to show that this is a
> > > > > >>> legitimate
> > > > > >>>>> use
> > > > > >>>>>>>> case.
> > > > > >>>>>>>>
> > > > > >>>>>>>> Since I don't see the concerns of the different use cases
> > > > > >>> changing,
> > > > > >>>>> it
> > > > > >>>>>>>> looks like we need to find a compromise to move forward. I
> > > > > >>> propose
> > > > > >>>>> that
> > > > > >>>>>>> we
> > > > > >>>>>>>> introduce a FINITE_VECTOR type and separately a
> > > > > >> FIXED_LENGTH_LIST
> > > > > >>>> or
> > > > > >>>>>>>> another VECTOR type that allows infinite and NaN values.
> > > > > >>>>>>>>
> > > > > >>>>>>>> Can we agree on a compromise? Is there a strong reason to
> > try
> > > > > >> to
> > > > > >>>> fit
> > > > > >>>>>>> these
> > > > > >>>>>>>> use cases into a single type? What if that means not
> making
> > > > > >>>> progress?
> > > > > >>>>>>>>
> > > > > >>>>>>>> Ryan
> > > > > >>>>>>>>
> > > > > >>>>>>>> On Mon, Oct 5, 2026 at 12:15 PM Philipp Fischbeck via dev
> <
> > > > > >>>>>>>> dev@parquet.apache.org> wrote:
> > > > > >>>>>>>>
> > > > > >>>>>>>>> Thinking about this more, I am also fine with pulling out
> > the
> > > > > >>>>>> “NaN/Inf”
> > > > > >>>>>>>>> requirement. Either into a type boolean (my preference),
> or
> > > > > >>>> putting
> > > > > >>>>>> no
> > > > > >>>>>>>>> restrictions on the Parquet level, allowing all IEEE 754
> > > > > >> float
> > > > > >>>>>> numbers.
> > > > > >>>>>>>> If
> > > > > >>>>>>>>> some top-level application wants to put additional
> > > > > >> constraints
> > > > > >>> on
> > > > > >>>>> the
> > > > > >>>>>>>> data
> > > > > >>>>>>>>> type, like Iceberg not allowing NaN/infinity, it can
> still
> > do
> > > > > >>> so
> > > > > >>>>> one
> > > > > >>>>>>>> layer
> > > > > >>>>>>>>> above. It seems like this was the biggest point of
> > discussion
> > > > > >>>>>>> remaining.
> > > > > >>>>>>>>> All these options work for me, and we should move forward
> > > > > >> with
> > > > > >>>> this
> > > > > >>>>>>>>> proposal.
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> Concerning additional element types like bf16, they can
> be
> > > > > >>>>> introduced
> > > > > >>>>>>> as
> > > > > >>>>>>>>> Parquet types (at least in the current LIST-based spec
> > > > > >>> proposal).
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> Concerning Will’s suggestion around representing VECTOR
> as
> > a
> > > > > >>>>> variable
> > > > > >>>>>>>>> length list with a MIN/MAX_LENGTH metadata restriction: I
> > > > > >> think
> > > > > >>>>> that
> > > > > >>>>>> a
> > > > > >>>>>>>>> dedicated logical VECTOR type still provides enough
> > benefits
> > > > > >> by
> > > > > >>>>>>>> describing
> > > > > >>>>>>>>> the semantics of the data, and allowing to capture future
> > > > > >>> changes
> > > > > >>>>>>> around
> > > > > >>>>>>>>> statistics. In that case, the length is better captured
> as
> > an
> > > > > >>>> exact
> > > > > >>>>>>>> length
> > > > > >>>>>>>>> for this type specifically. I’m also not sure about other
> > use
> > > > > >>>> cases
> > > > > >>>>>> for
> > > > > >>>>>>>>> MIN_LENGTH, so introducing a new type constraint system
> > just
> > > > > >>> for
> > > > > >>>>> the
> > > > > >>>>>>>>> MIN_LENGTH=MAX_LENGTH case for vectors doesn’t have
> enough
> > > > > >>> value
> > > > > >>>> in
> > > > > >>>>>> my
> > > > > >>>>>>>>> view.
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> Best,
> > > > > >>>>>>>>> Philipp
> > > > > >>>>>>>>>
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> On Thu, Oct 1, 2026 at 9:00 AM Will Edwards via dev <
> > > > > >>>>>>>>> dev@parquet.apache.org>
> > > > > >>>>>>>>> wrote:
> > > > > >>>>>>>>>
> > > > > >>>>>>>>>> +1 Sebastian, a vector type suggests usage beyond
> > > > > >> normalized
> > > > > >>>>>>> embeddings
> > > > > >>>>>>>>> and
> > > > > >>>>>>>>>> engines really aren't going to be able to elide all the
> > > > > >>> checks
> > > > > >>>>>>> anyway.
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>> Antoine, you suggested we explore metadata solutions too
> > :)
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>> Today, a database supporting CHAR(N) or VARCHAR(n) can't
> > > > > >>>> express
> > > > > >>>>>> this
> > > > > >>>>>>>> in
> > > > > >>>>>>>>>> Parquet.  Engines can write opaque blobs they understand
> > in
> > > > > >>> the
> > > > > >>>>>>>> metadata
> > > > > >>>>>>>>>> (e.g. Arrow does) so the extra type info round-trips,
> but
> > > > > >>> this
> > > > > >>>>>> isn't
> > > > > >>>>>>>>>> standardized, portable, nor interoperable.
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>> So perhaps standardizing constraint metadata or new
> Thrift
> > > > > >>>> fields
> > > > > >>>>>> (my
> > > > > >>>>>>>>> pref)
> > > > > >>>>>>>>>> can generically support VECTOR(N) whilst also being more
> > > > > >>>> broadly
> > > > > >>>>>>> useful
> > > > > >>>>>>>>> for
> > > > > >>>>>>>>>> other use cases and data types?
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>> Imagine metadata or Thrift fields to tag the MIN_LENGTH
> > and
> > > > > >>>>>>> MAX_LENGTH
> > > > > >>>>>>>>> for
> > > > > >>>>>>>>>> Parquet's variable-length physcial types.  These would
> > > > > >> count
> > > > > >>>>>> logical
> > > > > >>>>>>>>> units
> > > > > >>>>>>>>>> (e.g. UTF8 code points for STRING) rather than bytes.
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>> Then one engine's VECTOR(N) can be written to Parquet as
> > > > > >>>> LIST(N)
> > > > > >>>>>> with
> > > > > >>>>>>>>>> MIN_LENGTH = MAX_LENGTH = N and then readers - even
> > > > > >> different
> > > > > >>>>>>> engines -
> > > > > >>>>>>>>> can
> > > > > >>>>>>>>>> pick up these constraints if they support doing so.  We
> > > > > >> don't
> > > > > >>>>>>> actually
> > > > > >>>>>>>>> need
> > > > > >>>>>>>>>> a logical type to express a VECTOR then.
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>> This also allows standard SQL's CHAR(N) and VARCHAR(N)
> to
> > > > > >> be
> > > > > >>>>>>> expressed
> > > > > >>>>>>>> on
> > > > > >>>>>>>>>> UTF8 columns etc.
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>> This also seems generally useful in the future as we
> > > > > >> approach
> > > > > >>>>>>>> expressing
> > > > > >>>>>>>>>> other tensor data shapes and types.
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>> This won't change the mechanics: the storage type
> doesn't
> > > > > >>>> enforce
> > > > > >>>>>>> these
> > > > > >>>>>>>>>> constraints and engines must check them. It's previously
> > > > > >>>>>> established
> > > > > >>>>>>>> that
> > > > > >>>>>>>>>> performance is just a software problem and there is no
> > > > > >>>>> fundamental
> > > > > >>>>>>>>>> performance advantage to the fixed size types in
> > > > > >> Dremel-like
> > > > > >>>>>> formats
> > > > > >>>>>>>> like
> > > > > >>>>>>>>>> Parquet.
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>> Best,
> > > > > >>>>>>>>>> Will
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>> On Wed, 30 Sept 2026 at 16:08, Sebastian Baunsgaard <
> > > > > >>>>>>>>> baunsgaard@apache.org
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>> wrote:
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>>> After listening in on the last meeting Wednesday, I
> have
> > > > > >>> been
> > > > > >>>>>>>> thinking
> > > > > >>>>>>>>>>> about this quite a bit, and thought I would share my 2
> > > > > >>> cents.
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>> I suggest not requiring VECTOR elements to be finite,
> as
> > > > > >>>>> proposed
> > > > > >>>>>>> in
> > > > > >>>>>>>>> the
> > > > > >>>>>>>>>> PR
> > > > > >>>>>>>>>>> <https://github.com/apache/parquet-format/pull/624>.
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>> While Divjot and others note the specialization for
> > > > > >> UTF-8,
> > > > > >>>>>> Parquet
> > > > > >>>>>>>>>> already
> > > > > >>>>>>>>>>> defines DOUBLE and FLOAT as IEEE encodings
> > > > > >>>>>>>>>>> <
> > > > > >>>>>>>>
> > > > > >>>>
> > > > https://github.com/apache/parquet-format/blob/master/README.md#types
> > > > > >>>>>>>>>> .
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>> Making NaN or +/-Inf illegal inside VECTOR would be a
> new
> > > > > >>>>>>> restriction
> > > > > >>>>>>>>> on
> > > > > >>>>>>>>>>> those same underlying physical types. Scalar
> FLOAT/DOUBLE
> > > > > >>>>> columns
> > > > > >>>>>>>> have
> > > > > >>>>>>>>>>> never had such a rule.
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>> It would also not match how the systems that actually
> > > > > >>> produce
> > > > > >>>>>> these
> > > > > >>>>>>>>>> vectors
> > > > > >>>>>>>>>>> treat floats. Most of them use IEEE floats, e.g. Spark,
> > > > > >>>> DuckDB,
> > > > > >>>>>>>> NumPy,
> > > > > >>>>>>>>>>> PyTorch, TensorFlow, or lower level CUDA, oneMKL, and
> > > > > >>>> OpenBLAS.
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>> pgvector's "all elements must be finite" rule is an
> > > > > >> insert
> > > > > >>>>>> policy.
> > > > > >>>>>>>>> While
> > > > > >>>>>>>>>> I
> > > > > >>>>>>>>>>> think Parquet as a general-purpose format should allow
> > > > > >> them
> > > > > >>>> out
> > > > > >>>>>> of
> > > > > >>>>>>>> the
> > > > > >>>>>>>>>> box.
> > > > > >>>>>>>>>>> It is a user system decision to disallow writing these
> > > > > >>> values
> > > > > >>>>> or
> > > > > >>>>>> to
> > > > > >>>>>>>>> throw
> > > > > >>>>>>>>>>> errors when encountering them.
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>> A type-level ban also does not keep Inf/NaN out of
> vector
> > > > > >>>> math.
> > > > > >>>>>>>> Finite
> > > > > >>>>>>>>>>> coordinates still overflow. For instance: magnitude =
> > > > > >>>>>> ||[FLOAT_MAX,
> > > > > >>>>>>>>>>> FLOAT_MAX]|| where every coordinate is finite overflows
> > > > > >> to
> > > > > >>>>> +Inf.
> > > > > >>>>>>>>> PyTorch
> > > > > >>>>>>>>>>> documents the same overflow: tensor([1e20,
> 1e20]).norm()
> > > > > >> is
> > > > > >>>> inf
> > > > > >>>>>> in
> > > > > >>>>>>>>> fp32.
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>> A generated MAGNITUDE(vec) column would then be Inf
> while
> > > > > >>> the
> > > > > >>>>>>> VECTOR
> > > > > >>>>>>>>>> itself
> > > > > >>>>>>>>>>> was required to be finite. The ban only forbids storing
> > > > > >>>>> specials
> > > > > >>>>>> in
> > > > > >>>>>>>> the
> > > > > >>>>>>>>>>> coordinates, and does not prevent even simple
> downstream
> > > > > >>>>>>> computations
> > > > > >>>>>>>>>> from
> > > > > >>>>>>>>>>> producing them.
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>> I suggest keeping VECTOR as the type and add an
> optional
> > > > > >>> bool
> > > > > >>>>>>>>>>> VectorType.finite (default false). Engines that want
> > > > > >>>>>> pgvector-like
> > > > > >>>>>>>>>>> rejection can require the flag = true. Engines that use
> > > > > >>>> NumPy /
> > > > > >>>>>>>>> PyTorch /
> > > > > >>>>>>>>>>> Spark arrays can keep representable IEEE values. For
> the
> > > > > >>> same
> > > > > >>>>>>>> reason, I
> > > > > >>>>>>>>>> do
> > > > > >>>>>>>>>>> not favor adding yet another type called FINITE_VECTOR.
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>> Best,
> > > > > >>>>>>>>>>> Sebastian
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>> On Tue, Sep 29, 2026 at 2:03 PM Divjot Arora via dev <
> > > > > >>>>>>>>>>> dev@parquet.apache.org>
> > > > > >>>>>>>>>>> wrote:
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>>> I agree with moving forward with the proposal and
> > > > > >>> renaming
> > > > > >>>>>> VECTOR
> > > > > >>>>>>>> to
> > > > > >>>>>>>>>>>> FINITE_VECTOR or something
> > > > > >>>>>>>>>>>> similar if others feel that better communicates its
> > > > > >>>> intent. I
> > > > > >>>>>> see
> > > > > >>>>>>>>> value
> > > > > >>>>>>>>>>> in
> > > > > >>>>>>>>>>>> having a vector type that guarantees
> > > > > >>>>>>>>>>>> its values are mathematically-correct vectors for a
> few
> > > > > >>>>>> reasons:
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>> 1. As others have mentioned, this is similar to the
> > > > > >>> STRING
> > > > > >>>>>>> logical
> > > > > >>>>>>>>> type
> > > > > >>>>>>>>>>>> mandating UTF-8.
> > > > > >>>>>>>>>>>> 2. It is not possible to rely on statistics for
> > > > > >> detecting
> > > > > >>>> NaN
> > > > > >>>>>> or
> > > > > >>>>>>>>> +-inf
> > > > > >>>>>>>>>>>> values as all statistics are optional.
> > > > > >>>>>>>>>>>> To my knowledge, there is no precedent for requiring
> > > > > >> that
> > > > > >>>>>> writers
> > > > > >>>>>>>>> emit
> > > > > >>>>>>>>>>>> stats for specific logical types.
> > > > > >>>>>>>>>>>> 3. Without statistics, NaN/inf values could flow
> > > > > >> through
> > > > > >>>> the
> > > > > >>>>>>> engine
> > > > > >>>>>>>>> and
> > > > > >>>>>>>>>>>> block them for specific functionality.
> > > > > >>>>>>>>>>>> This seems to be Lance's behavior. However, I don't
> > > > > >> think
> > > > > >>>>> this
> > > > > >>>>>> is
> > > > > >>>>>>>> as
> > > > > >>>>>>>>>> easy
> > > > > >>>>>>>>>>>> as it sounds and Lance has run
> > > > > >>>>>>>>>>>> into several bugs due to this ([1] [2] for examples).
> > > > > >>> Their
> > > > > >>>>>>> Python
> > > > > >>>>>>>>> [3]
> > > > > >>>>>>>>>>> and
> > > > > >>>>>>>>>>>> Rust [4] APIs default to erroring on
> > > > > >>>>>>>>>>>> vectors that contain NaN values.
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>> I agree that Parquet is a general-purpose format, but
> I
> > > > > >>>> don't
> > > > > >>>>>>> think
> > > > > >>>>>>>>>> this
> > > > > >>>>>>>>>>>> implies the lack of domain-specific types.
> > > > > >>>>>>>>>>>> Instead, this broadens the use cases where Parquet
> > > > > >> would
> > > > > >>> be
> > > > > >>>>>>>>> considered
> > > > > >>>>>>>>>> a
> > > > > >>>>>>>>>>>> good fit. I also fully agree with
> > > > > >>>>>>>>>>>> continuing work on a general fixed-size list type if
> we
> > > > > >>>>> believe
> > > > > >>>>>>>> there
> > > > > >>>>>>>>>> are
> > > > > >>>>>>>>>>>> non-vector use cases that would benefit
> > > > > >>>>>>>>>>>> from such guarantees at the storage layer.
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>> [1] https://github.com/lance-format/lance/pull/3648
> > > > > >>>>>>>>>>>> [2] https://github.com/lance-format/lance/issues/8607
> > > > > >>> (see
> > > > > >>>>>>> point 2
> > > > > >>>>>>>>> in
> > > > > >>>>>>>>>>>> "root
> > > > > >>>>>>>>>>>> cause")
> > > > > >>>>>>>>>>>> [3]
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>
> > > > > >>>>>>>>
> > > > > >>>>>>>
> > > > > >>>>>>
> > > > > >>>>>
> > > > > >>>>
> > > > > >>>
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://github.com/lancedb/lancedb/blob/0327618a62483feb6dca2336549845e5887cd48f/python/python/lancedb/table.py#L4807-L4911
> > > > > >>>>>>>>>>>> [4]
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>
> > > > > >>>>>>>>
> > > > > >>>>>>>
> > > > > >>>>>>
> > > > > >>>>>
> > > > > >>>>
> > > > > >>>
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://github.com/lancedb/lancedb/blob/0327618a62483feb6dca2336549845e5887cd48f/rust/lancedb/src/table/add_data.rs#L105-L110
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>> On Mon, Sep 28, 2026 at 6:19 PM Philipp Fischbeck via
> > > > > >>> dev <
> > > > > >>>>>>>>>>>> dev@parquet.apache.org> wrote:
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>>> I also favor moving forward with the proposal as is.
> > > > > >> I
> > > > > >>>> see
> > > > > >>>>>>> strong
> > > > > >>>>>>>>>> value
> > > > > >>>>>>>>>>>> and
> > > > > >>>>>>>>>>>>> use cases for the numeric vector type, and I agree
> > > > > >> with
> > > > > >>>> the
> > > > > >>>>>>>>>>> restrictions
> > > > > >>>>>>>>>>>>> around NaN/Inf values and element types in general.
> > > > > >>>>> Precedent
> > > > > >>>>>>>>> exists
> > > > > >>>>>>>>>> in
> > > > > >>>>>>>>>>>>> other types that restrict which physical type values
> > > > > >>> are
> > > > > >>>>>>>> logically
> > > > > >>>>>>>>>>> valid
> > > > > >>>>>>>>>>>>> (e.g., the geospatial types needing to be
> > > > > >> WKB-encoded).
> > > > > >>>> A
> > > > > >>>>>>>> general
> > > > > >>>>>>>>>>>>> fixed-size list also has value, but I think it makes
> > > > > >>>> sense
> > > > > >>>>> to
> > > > > >>>>>>>>>> separate
> > > > > >>>>>>>>>>>> the
> > > > > >>>>>>>>>>>>> two types. The semantics are relevant, and I can see
> > > > > >>> how
> > > > > >>>>> the
> > > > > >>>>>>> two
> > > > > >>>>>>>>>> types
> > > > > >>>>>>>>>>>>> might diverge further in the future regarding
> > > > > >>> statistics
> > > > > >>>> or
> > > > > >>>>>>>>>> encodings.
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>> I would be okay with renaming the vector type, but it
> > > > > >>>> seems
> > > > > >>>>>>> like
> > > > > >>>>>>>>> the
> > > > > >>>>>>>>>>>> VECTOR
> > > > > >>>>>>>>>>>>> name is already used across other engines mentioned
> > > > > >> in
> > > > > >>>> this
> > > > > >>>>>>>> thread.
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>> Best,
> > > > > >>>>>>>>>>>>> Philipp
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>> On Sat, Sep 26, 2026 at 4:37 PM Gang Wu <
> > > > > >>>> ustcwg@gmail.com>
> > > > > >>>>>>>> wrote:
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>> To separate orthogonal concerns, I feel we should
> > > > > >>> treat
> > > > > >>>>> the
> > > > > >>>>>>>>> general
> > > > > >>>>>>>>>>>>>> FixedSizeList as the underlying physical layout,
> > > > > >> and
> > > > > >>>>>> separate
> > > > > >>>>>>>> it
> > > > > >>>>>>>>>> from
> > > > > >>>>>>>>>>>> the
> > > > > >>>>>>>>>>>>>> discussion of a specialized VECTOR (or broader ML
> > > > > >>>> tensor)
> > > > > >>>>>>>> logical
> > > > > >>>>>>>>>>> type.
> > > > > >>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>> Currently, the immediate need is to define a
> > > > > >>>> specialized
> > > > > >>>>>>> VECTOR
> > > > > >>>>>>>>>> type
> > > > > >>>>>>>>>>> so
> > > > > >>>>>>>>>>>>>> that different engines and formats have a common
> > > > > >>>> semantic
> > > > > >>>>>>>>> contract
> > > > > >>>>>>>>>>> for
> > > > > >>>>>>>>>>>>>> interoperability.
> > > > > >>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>> If the main point of contention is just the
> > > > > >>>> NaN/Inf/null
> > > > > >>>>>>>>>> limitations,
> > > > > >>>>>>>>>>>>> maybe
> > > > > >>>>>>>>>>>>>> we can resolve this by adding a qualification to
> > > > > >> the
> > > > > >>>> name
> > > > > >>>>>>> (like
> > > > > >>>>>>>>>>> Andrew
> > > > > >>>>>>>>>>>>>> suggested, e.g., VECTOR_EMBEDDING or FINITE_VECTOR)
> > > > > >>> to
> > > > > >>>>>> avoid
> > > > > >>>>>>>>>>> confusion
> > > > > >>>>>>>>>>>>> with
> > > > > >>>>>>>>>>>>>> broader ML use cases. Alternatively, we could
> > > > > >>>> introduce a
> > > > > >>>>>>>> boolean
> > > > > >>>>>>>>>>> flag
> > > > > >>>>>>>>>>>>>> (e.g., optional bool VectorType.finite) as proposed
> > > > > >>>>>> earlier.
> > > > > >>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>> On Fri, Sep 25, 2026 at 3:52 AM Andrew Lamb <
> > > > > >>>>>>>>>> andrewlamb11@gmail.com>
> > > > > >>>>>>>>>>>>>> wrote:
> > > > > >>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>> I am a little confused about the points of
> > > > > >>>> disagreement
> > > > > >>>>>>> here.
> > > > > >>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>> Is it that the name `VECTOR` is too general for
> > > > > >> the
> > > > > >>>>>>> specific
> > > > > >>>>>>>>> use
> > > > > >>>>>>>>>>> case
> > > > > >>>>>>>>>>>>> of
> > > > > >>>>>>>>>>>>>>> fixed length arrays with non-finite values used
> > > > > >> for
> > > > > >>>>>> vector
> > > > > >>>>>>>>>>>> embeddings?
> > > > > >>>>>>>>>>>>>> If
> > > > > >>>>>>>>>>>>>>> so, maybe we could add a qualification to the
> > > > > >> name
> > > > > >>>>>>>>>> (FINITE_VECTOR?
> > > > > >>>>>>>>>>>>>>> FEATURE_VECTOR? VECTOR_EMBEDDING?)
> > > > > >>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>> It seems like there is precedent for defining
> > > > > >>> logical
> > > > > >>>>>> types
> > > > > >>>>>>>>> that
> > > > > >>>>>>>>>>>>> specify
> > > > > >>>>>>>>>>>>>>> certain physical types and restrict their
> > > > > >> contents
> > > > > >>> to
> > > > > >>>>>> help
> > > > > >>>>>>>>>> readers
> > > > > >>>>>>>>>>>>>>> interpret meaning, for example string[1] or JSON;
> > > > > >>>>>> Defining
> > > > > >>>>>>>> one
> > > > > >>>>>>>>>> for
> > > > > >>>>>>>>>>>>>> vector
> > > > > >>>>>>>>>>>>>>> embeddings seems reasonable enough to me.
> > > > > >>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>> Andrew
> > > > > >>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>> [1]:
> > > > > >>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>
> > > > > >>>>>>>>
> > > > > >>>>>>>
> > > > > >>>>>>
> > > > > >>>>>
> > > > > >>>>
> > > > > >>>
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#string
> > > > > >>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>> On Thu, Sep 24, 2026 at 3:34 PM Daniel Weeks <
> > > > > >>>>>>>>> dweeks@apache.org>
> > > > > >>>>>>>>>>>>> wrote:
> > > > > >>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>> Well, Parquet is not a single-purpose format.
> > > > > >>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>> Vector stores can choose whatever policy fits
> > > > > >>>> their
> > > > > >>>>>>>> usage,
> > > > > >>>>>>>>>> but
> > > > > >>>>>>>>>>>>> we're
> > > > > >>>>>>>>>>>>>>>>> talking about a general-purpose file format
> > > > > >>> that
> > > > > >>>>> aims
> > > > > >>>>>>> to
> > > > > >>>>>>>> be
> > > > > >>>>>>>>>>>> broadly
> > > > > >>>>>>>>>>>>>>>>> applicable.
> > > > > >>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>> I agree that the format is intended to be
> > > > > >> general
> > > > > >>>>>>> purpose,
> > > > > >>>>>>>>> but
> > > > > >>>>>>>>>>>> types
> > > > > >>>>>>>>>>>>>>> within
> > > > > >>>>>>>>>>>>>>>> the format define specific representation and
> > > > > >>>>> semantics
> > > > > >>>>>>> to
> > > > > >>>>>>>>>>> achieve
> > > > > >>>>>>>>>>>>> that
> > > > > >>>>>>>>>>>>>>>> goal.
> > > > > >>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>> "Vector" has a more general sense than ML and
> > > > > >>>> it's
> > > > > >>>>>>> quite
> > > > > >>>>>>>>>>> trivial
> > > > > >>>>>>>>>>>> to
> > > > > >>>>>>>>>>>>>>>>> produce vectors with non-finite values using
> > > > > >>> e.g.
> > > > > >>>>>> NumPy
> > > > > >>>>>>>> or
> > > > > >>>>>>>>>>> Pandas
> > > > > >>>>>>>>>>>>> (or
> > > > > >>>>>>>>>>>>>>>>> any other piece of dataframe or scientific
> > > > > >>>>> computing
> > > > > >>>>>>>>>> software).
> > > > > >>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>> The justification for restricting non-finite
> > > > > >>>> values
> > > > > >>>>> is
> > > > > >>>>>>>>>> directly
> > > > > >>>>>>>>>>>>>> related
> > > > > >>>>>>>>>>>>>>> to
> > > > > >>>>>>>>>>>>>>>> the semantic meaning of a vector.  Vectors have
> > > > > >>>>>> specific
> > > > > >>>>>>>>>>>>>> characteristics
> > > > > >>>>>>>>>>>>>>>> (dimension, magnitude, direction, etc).
> > > > > >>> Non-finite
> > > > > >>>>>>> values
> > > > > >>>>>>>>> are
> > > > > >>>>>>>>>>>>> invalid
> > > > > >>>>>>>>>>>>>>> and
> > > > > >>>>>>>>>>>>>>>> break those characteristics.  I'd liken it to
> > > > > >>>> storing
> > > > > >>>>>>>>> non-utf8
> > > > > >>>>>>>>>>>> values
> > > > > >>>>>>>>>>>>>> in
> > > > > >>>>>>>>>>>>>>> a
> > > > > >>>>>>>>>>>>>>>> string and still calling it a string.
> > > > > >>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>> That also differentiates a vector from a fixed
> > > > > >>>> length
> > > > > >>>>>>> list,
> > > > > >>>>>>>>>> which
> > > > > >>>>>>>>>>>> is
> > > > > >>>>>>>>>>>>>> more
> > > > > >>>>>>>>>>>>>>>> general and does not carry the same semantic
> > > > > >>>> meaning.
> > > > > >>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>> I also don't think we need to bundle all
> > > > > >>> similarly
> > > > > >>>>>> shaped
> > > > > >>>>>>>> use
> > > > > >>>>>>>>>>> cases
> > > > > >>>>>>>>>>>>>> into
> > > > > >>>>>>>>>>>>>>>> the same solution (e.g. flxed size list).
> > > > > >> Vector
> > > > > >>>> is
> > > > > >>>>> a
> > > > > >>>>>>> well
> > > > > >>>>>>>>>>>>> established
> > > > > >>>>>>>>>>>>>>>> data type at this point and aligns with the
> > > > > >>> general
> > > > > >>>>>>> purpose
> > > > > >>>>>>>>>>> nature
> > > > > >>>>>>>>>>>> of
> > > > > >>>>>>>>>>>>>> the
> > > > > >>>>>>>>>>>>>>>> format (many other formats and databases
> > > > > >> natively
> > > > > >>>>>> support
> > > > > >>>>>>>>>> this).
> > > > > >>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>> -Dan
> > > > > >>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>> On Thu, Sep 24, 2026 at 11:55 AM Antoine
> > > > > >> Pitrou <
> > > > > >>>>>>>>>>>> antoine@python.org>
> > > > > >>>>>>>>>>>>>>>> wrote:
> > > > > >>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>> Le 24/09/2026 à 20:11, Russell Spitzer a
> > > > > >> écrit
> > > > > >>> :
> > > > > >>>>>>>>>>>>>>>>>> I think something I haven't seen yet is the
> > > > > >>>> clear
> > > > > >>>>>>>>> use-case
> > > > > >>>>>>>>>>> for
> > > > > >>>>>>>>>>>>>>>> non-finite
> > > > > >>>>>>>>>>>>>>>>>> values within a vector type. There has
> > > > > >> been a
> > > > > >>>> lot
> > > > > >>>>>> of
> > > > > >>>>>>>>>>>> theoretical
> > > > > >>>>>>>>>>>>>>>>> discussion
> > > > > >>>>>>>>>>>>>>>>>> here about use cases but what toolings
> > > > > >>>> currently
> > > > > >>>>>>>> produce
> > > > > >>>>>>>>>>> these
> > > > > >>>>>>>>>>>>>>> vectors
> > > > > >>>>>>>>>>>>>>>>> and
> > > > > >>>>>>>>>>>>>>>>>> how do they function in downstream engines?
> > > > > >>> I'd
> > > > > >>>>>> like
> > > > > >>>>>>> to
> > > > > >>>>>>>>>> limit
> > > > > >>>>>>>>>>>>>>>> introducing
> > > > > >>>>>>>>>>>>>>>>>> types that have a wide range of values
> > > > > >> which
> > > > > >>>> are
> > > > > >>>>>>> simply
> > > > > >>>>>>>>> are
> > > > > >>>>>>>>>>>>> failure
> > > > > >>>>>>>>>>>>>>>> modes
> > > > > >>>>>>>>>>>>>>>>>> of different flavors
> > > > > >>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>> "Vector" has a more general sense than ML and
> > > > > >>>> it's
> > > > > >>>>>>> quite
> > > > > >>>>>>>>>>> trivial
> > > > > >>>>>>>>>>>> to
> > > > > >>>>>>>>>>>>>>>>> produce vectors with non-finite values using
> > > > > >>> e.g.
> > > > > >>>>>> NumPy
> > > > > >>>>>>>> or
> > > > > >>>>>>>>>>> Pandas
> > > > > >>>>>>>>>>>>> (or
> > > > > >>>>>>>>>>>>>>>>> any other piece of dataframe or scientific
> > > > > >>>>> computing
> > > > > >>>>>>>>>> software).
> > > > > >>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>> It's not like the general concept of a vector
> > > > > >>> is
> > > > > >>>> a
> > > > > >>>>>> IETF
> > > > > >>>>>>>>>>> standard
> > > > > >>>>>>>>>>>> or
> > > > > >>>>>>>>>>>>>>>>> anything, so analogies with JSON or UTF8
> > > > > >> (where
> > > > > >>>> the
> > > > > >>>>>>>> values
> > > > > >>>>>>>>>> are
> > > > > >>>>>>>>>>>>>> expected
> > > > > >>>>>>>>>>>>>>>>> to be valid according to the respective
> > > > > >> specs)
> > > > > >>> do
> > > > > >>>>> not
> > > > > >>>>>>>>> apply.
> > > > > >>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>> This discussion would IMHO have a better
> > > > > >> chance
> > > > > >>>> of
> > > > > >>>>>>>>> converging
> > > > > >>>>>>>>>>> if
> > > > > >>>>>>>>>>>> we
> > > > > >>>>>>>>>>>>>>>>> separated the two orthogonal concerns of:
> > > > > >>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>> 1. representing fixed-length list/vector data
> > > > > >>> in
> > > > > >>>>>>> Parquet
> > > > > >>>>>>>>>> (with
> > > > > >>>>>>>>>>> a
> > > > > >>>>>>>>>>>>>>>>> potential limitation to non-nested types)
> > > > > >>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>> 2. representing the information that a given
> > > > > >>>> column
> > > > > >>>>>> (or
> > > > > >>>>>>>>>> column
> > > > > >>>>>>>>>>>>> chunk,
> > > > > >>>>>>>>>>>>>>>>> etc.) does not contain non-finite values
> > > > > >>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>> ... both of which are useful on their own
> > > > > >>> right,
> > > > > >>>>> but
> > > > > >>>>>> do
> > > > > >>>>>>>> not
> > > > > >>>>>>>>>>> have
> > > > > >>>>>>>>>>>> to
> > > > > >>>>>>>>>>>>>> be
> > > > > >>>>>>>>>>>>>>>>> conflated and bundled together.
> > > > > >>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>> Regards
> > > > > >>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>> Antoine.
> > > > > >>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>
> > > > > >>>>>>>>
> > > > > >>>>>>>
> > > > > >>>>>>
> > > > > >>>>>
> > > > > >>>>
> > > > > >>>
> > > > > >>
> > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] Numeric Vector type: physical representation (outcome of focused call)

[Ryan Blue <bl...@apache.org>]

An annotation on list<float> is useful because:
- We want to correctly annotate what is being stored
- We want to unlock optimizations when each non-null list has the same
number of elements
- We want to have writer guarantees

I don't think that logical type compatibility is a good argument. We want
to avoid a case where we have to treat all binary like a string. And
because we are introducing this in table formats after standardizing it
here, we can do that reliably.

As I said in my last email, I think there's a difference between finiteness
and general "invariants" you might want. Vectors aren't vectors if they
don't have a length and a direction. Other invariants don't work the same
way. A zero-length vector is still a valid vector and can be used, just not
for certain distance metrics.

I also want to clarify that when we looked into this more closely and
recognized the mismatch between the two separate use cases, I offered to
split out the finite vector work so that we didn't cause any headaches for
people who are interested in fixed-length list. It sounds like not
splitting out the use cases is causing a lot of headache. Vectors for our
use case need a finiteness guarantee at write time, and fixed-length lists
need to store NaN and other values.

It's okay to have two types and we should for two different use cases.
Otherwise, we are forcing one to make an unnecessary compromise. I don't
want to force everyone to exclude NaN and infinite values from fixed-length
lists. Seems reasonable that we should also be able to have vectors that
exclude those values, since that is how they are used and we have to also
guarantee it for exchanging data with other existing databases.

Ryan

On Mon, Oct 19, 2026 at 3:12 AM Will Edwards via dev <de...@parquet.apache.org>
wrote:

> It reads well. Three small fixes:
>
> 1. **"back in May or so"** is ambiguous. The fixed-size-list-for-tensors
> framing is from Rok's opening post in May 2024. The May 2026 design doc was
> already mostly about performance and layout. "Originally" avoids anyone
> correcting the date.
> 2. **"Spark's sparse feature vectors routinely use NaN"** isn't quite
> right. The NaN-for-missing behaviour comes from `VectorAssembler` and
> `Imputer` and applies to Spark ML vectors generally, dense or sparse.
> "Spark's feature vectors, dense and sparse," fixes it.
> 3. There are typos: "Iirc" should be "IIRC", and "Spark, BigQuery and
> DuckDB and pretty much…" has a doubled "and".
>
> The last sentence of the invariants paragraph could be clearer. "Although
> that doesn't mean readers shouldn't check on read" has a double negative
> and is easy to misread. "Though careful readers will still check on read"
> is simpler.
>
> With those applied:
>
> ---
>
> Hi all,
>
> The scope of this discussion keeps moving :). IIRC it originally was about
> fixed-size lists for ML tensors, became a performance question, and is now
> about whether vectors are mathematically valid. Each step made sense, but I
> don't think we have consensus on where we've ended up?
>
> I think there is much useful work in what we left behind. Many ML vectors
> aren't embeddings for search. E.g. Spark's feature vectors, dense and
> sparse, routinely use NaN to mark missing values, and quantised formats
> pack codes with scales in ways Parquet can't describe today. ML-oriented
> formats like Lance are moving towards tensors, quantisation and indexes. A
> type that told engines how to interpret that kind of data would enable new
> things. An annotation on LIST<FLOAT> doesn't.
>
> My big concern with the current focus on embeddings needing a logical type
> is compatibility. Today LIST<FLOAT> is the common format for embeddings,
> and Spark, BigQuery, DuckDB and pretty much every other library I can think
> of offer distance functions on plain lists, so any engine can use any file.
> If engines start requiring a VECTOR annotation, existing files and anything
> rewritten by a tool that doesn't know the type stop working. If they don't
> require it, the annotation adds nothing. We've seen this before with
> strings written without the UTF8 annotation, which engines still have to
> cope with years later.
>
> On invariants, I agree some users may want the writer to check what they
> write as they write it, but that can be a writer option rather than a type.
> Readers that want evidence could get it from extra statistics or metadata,
> though careful readers will still check on read.
>
> If a finite fixed-length vector contract is useful to table formats,
> perhaps Iceberg is the place for it, with Parquet storing plain lists. I'd
> raise the same worry there, though: if tables start rejecting untagged
> columns, we've just moved the split up a layer.
>
> Best,
> Will
>
> On Sat, 17 Oct 2026 at 00:10, Ryan Blue <bl...@apache.org> wrote:
>
> > I agree with much of what Dan said, but I'll add some to this thread as
> > well.
> >
> > First, I want to address the arguments that were pointed out:
> > 1. Other vector databases have finiteness requirements
> > 2. "Vectors are the hot stuff right now" -- let's jump on the bandwagon
> > 3. A general solution will take too long, finiteness will make faster
> > progress
> > 4. To give a guarantee to readers that no INF/NaN/NULL will be in the
> > vector
> >
> > I don't think these represent our actual motivation or position, which
> > makes them function as a straw-man. This is probably not what was
> > intended, but I want to make sure that the misunderstanding
> > doesn't continue. For one thing, we aren't saying that we need finite
> > vectors because they are "hot stuff".
> >
> > We want to build tools that use them on top of Parquet. I think it should
> > also be clear that we aren't advocating for finite elements blindly just
> > because other engines do (cargo cult design!). There are real
> > interoperability issues if we choose to go a different direction so this
> is
> > a "Chesterton's Fence" situation. It's important to understand why
> > those databases
> > made that choice. I've also pointed this out as confirmation that we're
> > going down a reasonable and established path.
> >
> > The third argument sounds like a simple misunderstanding. We're pursuing
> a
> > logical type right now because that's forward compatible and we want to
> > label the data properly and set expectations for the type. Physical type
> > changes break forward compatibility and need to be released in a new
> format
> > version . I think this is the source of the confusion since it has been
> in
> > the same discussion, but the choice about finiteness is separate.
> >
> > That brings us to the last argument, whether there is value in writers
> > participating in guarantees. I think there's value in adding requirements
> > for writers. It's up to readers how they want to use those requirements,
> > but a writer requirement allows a policy of "garbage in garbage out"
> rather
> > than being forced to handle bad data. Why would a reader need to run
> > validation on every element just in case a writer had a bug? And why
> should
> > the potential for writer bugs mean that it has to be the reader's
> problem?
> > If a writer produces bad data, we can absolutely reject it and tell the
> > writer to fix the bug.
> >
> > I'm also puzzled by the assertion that writer guarantees have value, but
> > only if there are other, more specific guarantees in addition. This seems
> > to cede the point that a contract between readers and writers makes
> sense,
> > but only if it is a _specific_ contract. But the proposed additions don't
> > make sense in the same way.
> >
> > Take, for example, the suggestion to also disallow zero-length vectors.
> At
> > a glance this seems similar, but I think it assumes a specific context
> and
> > is better suited to be a constraint for some use cases rather than part
> of
> > what is considered valid for the type. That's because zero-length vectors
> > are valid vectors with a length. You can calculate Euclidean or Manhattan
> > distance and use them in many contexts. They don't work if you're
> > normalizing to a length of 1, but it's a choice to normalize and use a
> dot
> > product to calculate cosine distance. And at this point, we're talking
> > about the usefulness of restrictions for the type, not a Platonic ideal.
> >
> > I think the restriction that makes the most sense is that vectors have a
> > length and can be compared in distance calculations. 0-length vectors are
> > valid because they point somewhere and have a length. We shouldn't
> restrict
> > uses to only cosine distance over normalized vectors. Those are use case
> > dependent choices and there are other valid ones. I also don't think that
> > the model used to produce a vector is part of the vector type, just like
> > the method used to produce a metric doesn't define its type: cosine
> > similarity produces values between -1 and 1, but we store it in a
> `float`,
> > not a `float(cosine)`.
> >
> > Why does it make sense to have a finiteness restriction as part of the
> > type? Because that's what a (mathematical) vector is. A simple search
> gives
> > a pretty common definition: "A vector is an object that has both
> magnitude
> > and a direction". If a dimension/element is NaN, null, or infinite, the
> > vector doesn't point anywhere and it doesn't have a magnitude. Not only
> can
> > it not be used, it doesn't meet the most basic definition of the type.
> >
> > Ryan
> >
> > On Thu, Oct 15, 2026 at 1:07 AM Jan Finis <jp...@gmail.com> wrote:
> >
> > > >
> > > >
> > > >
> > > Daniel,
> > >
> > > alright, so no physical optimizations intended.  We only want this for
> > > semantic reasons, not for any significant physical optimization. If
> we're
> > > all on the same page here, that is good 👍.
> > >
> > > To phrase my remaining problem with this more concretely:
> > > *Let's say we add this type. What cool new stuff can we now do that we
> > > couldn't do before?*
> > > That is the litmus test that a type has to pass to be useful IMHO. And
> > here
> > > I can't come up with a tangible use case that wasn't possible before.
> > > Especially if we one day have fixed size lists in Parquet, the
> remaining
> > > niche for this type becomes so small, that the added complexity isn't
> > worth
> > > the benefit IMHO.
> > >
> > > As I guess most people on here are active maintainers of a Parquet
> > > implementation: Concretely, what would you be using this type for if we
> > had
> > > it? Would you allow math only on fields tagged with this type? What if
> > that
> > > math yields INF or NaN? Not allowed?
> > >
> > > Especially if we also discuss having both VECTOR and FINITE_VECTOR. How
> > > will you guys wire this up with the type systems of your engines? Do
> your
> > > engines also have two types here? Do you type-decay a FINITE_VECTOR to
> a
> > > VECTOR once someone does a multiplication that could yield INF? By
> > itself,
> > > a FINITE_VECTOR sounds good. Holistically looked at from a type system
> > > perspective, it comes with a lot of complexity. Do the benefits really
> > > outweigh the complexity here? That's why I would rather have the bare
> > > minimum: A fixed size array (list) of floats or at least just a vector
> > > without finite. Then all these detailed questions go away.
> > >
> > > I'll take this one step further... Why is there any debate about this
> at
> > > > all? Why can't a type simply be added to the *specification* as an
> > > > administrative detail and be done?
> > >
> > >
> > > Andrew,
> > >
> > > Parquet is supposed to be a lingua franca of data. The specification is
> > the
> > > authoritative truth for this language and all implementations are
> > expected
> > > to implement as much as possible from this spec, so anything in the
> spec
> > > does causes implementation effort. So just adding anything to this spec
> > > without debate is not a good idea. It is even the other way around: We
> > need
> > > to have way higher rigor with the spec than we need to have with any
> > > implementations. Implementations can be changed, the spec needs to be
> > > stable.
> > >
> > > A tag-only type that doesn't come with any restrictions can be added
> with
> > > little effort. But here we want to tie restrictions to it (no NaN/Inf),
> > so
> > > we would expect implementations to enforce these. So there is some
> > > implementation effort expected here.
> > >
> > > Also, even if we only wanted a tag type, I would still argue that you
> > > shouldn't add those without a debate. What if A wants XML, B wants
> YAML,
> > C
> > > wants quaternions, D wants address, E wants phone number. If we just
> > added
> > > any types that are "just tags", Parquet logical types would just
> become a
> > > trash dump of any type that anyone ever thought to be useful. So
> rigorous
> > > assessment of each new type candidate is necessary.
> > >
> > > Finally, many engines couple parquet with another type system (e.g.,
> > Arrow
> > > or SQL). So even with a tag only type, you at least still have the
> > > implementation burden of wiring this up with your engine's type system.
> > >
> > > So, just because logical types are "just" logical, that doesn't mean we
> > can
> > > add them without any rigor.
> > >
> > >
> > > A few naive questions:
> > > > 1. Would a physical definition with NANs & Inf+/- allow efficient
> > > > encodings like ALP?
> > > > 2. If the physical representation allows values that make encodings
> > like
> > > > ALP impossible, can a tighter Logical type allow different encodings?
> > > > 3. Can we RLE or binpack a vector?
> > >
> > >
> > > The simple general answer here is that all these things do not require
> a
> > > logical type. Writers can already do any such optimization if the
> actual
> > > data contains no NaNs & Infs. All these physical optimizations just
> > require
> > > no actual NaNs/Infs to be present in a page, they don't need a
> guarantee
> > > that the whole file won't contain any.
> > >
> > > Cheers,
> > > Jan
> > >
> > >
> > >
> > >
> > > Am Do., 15. Okt. 2026 um 04:28 Uhr schrieb <ku...@gmail.com>:
> > >
> > > > Hi Everyone,
> > > >
> > > > Glad to see a Vector type being added. I have a few use cases I want
> to
> > > > pursue once we have an agreed upon definition.
> > > >
> > > > In regards to that definition discussion. I feel comfortable with the
> > > idea
> > > > of the semantic meaning Dan raises though I can’t shake a slight
> > > discomfort
> > > > (maybe FOMO) with limiting values in the physical definition.
> > > >
> > > > A few naive questions:
> > > > 1. Would a physical definition with NANs & Inf+/- allow efficient
> > > > encodings like ALP?
> > > > 2. If the physical representation allows values that make encodings
> > like
> > > > ALP impossible, can a tighter Logical type allow different encodings?
> > > > 3. Can we RLE or binpack a vector?
> > > >
> > > > Best Regards,
> > > > Kurtis C. Wright
> > > > > On Oct 14, 2026, at 12:45, Daniel Weeks <dw...@apache.org> wrote:
> > > > >
> > > > > Hey Jan,
> > > > >
> > > > > There's a lot that I disagree with respect to the framing of these
> > > > > arguments and how we should be thinking about introducing vector
> as a
> > > > > logical type.  There's a repeated tendency to let the physical
> > > > > representation or performance drive discussion around types as
> > opposed
> > > to
> > > > > what it represents.
> > > > >
> > > > > In parquet, establishing semantic meaning is the purpose of logical
> > > > types.
> > > > > The logical type separates how to interpret a value from its
> physical
> > > > > representation.  We can represent a timestamp with INT64 or even a
> > > string
> > > > > value, so why have a type for it? Even though a FIXED_SIZED_LIST
> and
> > > > VECTOR
> > > > > may have the exact same representation and performance, they have
> > > > different
> > > > > semantic meaning.  Databases and engines across the ecosystem
> already
> > > > > recognize this differentiation and have separate representations.
> > The
> > > > > choice to separate these isn't because the discussion is too
> tedious;
> > > > > they're fundamentally different things.
> > > > >
> > > > > The finite requirement is what makes a vector a reflection of what
> it
> > > > > represents.  As we've discussed in the thread, non-finite values in
> > the
> > > > > representation violate the properties of a vector and the value
> > ceases
> > > to
> > > > > be a vector.  On the other hand, a 0 vector is still a vector (you
> > can
> > > > > argue the utility of it, but it is a valid vector).
> Implementations
> > > can
> > > > > choose what they validate or how to handle execution errors; that's
> > an
> > > > > implementation concern, not a requirement.  Operations performed
> on a
> > > > > vector depend on the context in which they are used, but the
> > > > representation
> > > > > doesn't need to solve scenarios where that application doesn't make
> > > > sense.
> > > > >
> > > > > The guarantees or context needed to use a vector do not need to be
> > > > inherent
> > > > > in the type for it to be a useful type.  We don't need the currency
> > > used
> > > > > for decimal or whether a floating point is a normalized value as
> part
> > > of
> > > > > the type to make use of those types.  This conflates the use cases
> > with
> > > > the
> > > > > type system and unnecessarily attempts to draw the context into the
> > > type
> > > > > definition.  Similarly, you don't need to capture normalization,
> > model,
> > > > > etc. as part of the type for it to be useful.
> > > > >
> > > > > So the reason we want to add a vector type (or FINITE_VECTOR) is
> > > because
> > > > we
> > > > > want to store vectors with the semantic meaning and requirements
> > based
> > > on
> > > > > what they represent.
> > > > >
> > > > > -Dan
> > > > >
> > > > >
> > > > >> On Wed, Oct 14, 2026 at 1:22 AM Jan Finis <jp...@gmail.com>
> > wrote:
> > > > >>
> > > > >> A bit late to the party, but here we go:
> > > > >>
> > > > >> I have the strong sentiment that the FINITE_VECTOR type as it is
> > > > currently
> > > > >> being discussed will not be very useful and just increase
> > > implementation
> > > > >> complexity. The arguments for the type are all quite weak (details
> > see
> > > > >> below). The FINITE part doesn't seem to yield any value and is
> not a
> > > > useful
> > > > >> restriction by itself. The vector (i.e. fixed length) part should
> > > > rather be
> > > > >> generalized instead of making a one-off thing. In general, the
> > > existing
> > > > >> types are already working well for uncompressed vectors. The real
> > > > challenge
> > > > >> that would bring real benefits is supporting quantization. This is
> > > where
> > > > >> you get orders of magnitude improvements to storage space and
> > > processing
> > > > >> speed.
> > > > >>
> > > > >> Let's first discuss the FINITE part. So far, the only arguments I
> > have
> > > > read
> > > > >> for this are:
> > > > >>
> > > > >> 1. many other vector databases have it
> > > > >> 2. vectors are the hot stuff right now, so Parquet better boast
> > having
> > > > >> "native" type support for vectors
> > > > >> 3. by focussing on this common special case, we can make rapid
> > > progress,
> > > > >> while a general solution would take too long to specify
> > > > >> 4. to give a guarantee to readers that no INF/NaN/NULL will be in
> > the
> > > > >> vector
> > > > >>
> > > > >> the first two arguments are quite weak by themselves. Vector
> > databases
> > > > are
> > > > >> very specialized engines. Such engines can of course be very
> > > > opinionated on
> > > > >> their data types. A general format like Parquet must instead weigh
> > > > whether
> > > > >> a special type is worth the added complexity over a more general,
> > > widely
> > > > >> applicable type. In this case, it is IMHO not at all. And just
> > having
> > > a
> > > > >> type called "vector" for advertisement reasons to make Parquet
> > appear
> > > > more
> > > > >> modern seems a non-goal in the first place.
> > > > >>
> > > > >> The third "time to market" argument is valid for systems that
> don't
> > > > need a
> > > > >> stable interface. It is a very dangerous attitude for formats like
> > > > Parquet
> > > > >> that must be stable long term and should have a consolidated,
> > concise,
> > > > >> non-redundant type system. For such long term storage formats, we
> > need
> > > > to
> > > > >> take the time to build the right thing, as any shortcuts we take
> > will
> > > > stay
> > > > >> as blemishes in the format forever. If I read the discussion
> > > correctly,
> > > > >> this is actually the main argument as of now: We have found that
> it
> > is
> > > > hard
> > > > >> to align on how a fixed size list type should look like, so we
> want
> > to
> > > > go
> > > > >> the easier route of a specialized type. This seems like a
> > > > >> dangerous shortcut fueled by feature recency bias. Ask yourself:
> > Would
> > > > we
> > > > >> want this, if Parquet already had a fixed size list type?
> > > > >>
> > > > >> So finally, the only technical argument is the fourth one: "It
> > signals
> > > > >> semantics and ensures that no bad data lands in the format". I see
> > two
> > > > big
> > > > >> problems with this:
> > > > >>
> > > > >> 1. Readers *still* need to test for NaN / INF because Parquet
> files
> > > may
> > > > be
> > > > >> corrupt. Nothing can stop a malicious or buggy writer from
> > physically
> > > > >> writing NaN or INF. *So you save no single line of code in a
> reader.
> > > You
> > > > >> save no branches, as you still need to handle the INF/NaN case.*
> So
> > > this
> > > > >> doesn't help with performance at all. The only difference is, that
> > the
> > > > >> error message changes from "found an invalid INF value in the
> data"
> > to
> > > > >> "Your parquet file seems to be corrupt". It is actually quite the
> > > > opposite,
> > > > >> readers and writers get more complex having to handle this new
> type.
> > > > >>
> > > > >> If you now argue that it's not about performance, it's about
> defined
> > > > >> semantics and erroring out early in writers, then the second
> problem
> > > > >> arises:
> > > > >>
> > > > >> 2. The proposed no-NaN/INF/NULL guarantee is just one of multiple
> > > > >> guarantees that you need for sensible vector processing. If "no
> > > > >> INF/NaN/NULL" was the only guarantee you needed, I would agree
> that
> > > > >> FINITE_VECTOR is useful, but it is by no means the only guarantee
> > you
> > > > need,
> > > > >> instead, depending on the distance metric, you also need the
> > following
> > > > >> guarantees. E.g.:
> > > > >>
> > > > >>   - No vector may be the 0 vector, otherwise, e.g., cosine
> > similarity
> > > is
> > > > >>   undefined. Cosine similarity is often used as default, so this
> is
> > > > >> actually
> > > > >>   quite important in practice.
> > > > >>   - If we want to use a dot product, then all vectors need to
> > > guarantee
> > > > a
> > > > >>   length of 1. This would actually be quite useful for performance
> > > > >> reasons.
> > > > >>   - The vectors must come from the same model as the vector used
> for
> > > the
> > > > >>   search. Otherwise, the search yields a numeric result, but that
> > > result
> > > > >> has
> > > > >>   no sensible meaning.
> > > > >>
> > > > >> So, what use is a type that just guarantees part of the things you
> > > need
> > > > for
> > > > >> vector search but not others? You still need additional guarantees
> > in
> > > > >> addition to that type. What makes INF/NaN so important in
> comparison
> > > to,
> > > > >> say, "no 0 vector present", that it warrants an own type, while
> the
> > > > other
> > > > >> guarantees do not? If we really want this type as a basis for
> > > > cross-engine
> > > > >> sharing of vectors, then it needs to give *all* guarantees.
> > > > >>
> > > > >> I'm not against a specialized vector type if it really carries
> *all*
> > > > >> information to be useful by itself. If it carried all that
> > > information,
> > > > >> then a reading engine could indeed validate that only meaningful
> > > vector
> > > > >> searches are executed. But as long as it is just a half-baked
> thing,
> > > it
> > > > is
> > > > >> not useful by itself and just a burden on each implementer and
> > > > maintainer.
> > > > >> If we need extra information anyway, why not just stick to a
> general
> > > > type
> > > > >> like VECTOR (without FINITE).
> > > > >>
> > > > >> If you're a proponent of FINITE_VECTOR, ask yourself (or tell us)
> > what
> > > > >> advantage you will *really* get over just a (potentially fixed
> size)
> > > > List
> > > > >> of Float? As discussed, it's not performance. It's also not a
> > semantic
> > > > >> marker that is sufficient by itself. It's just a small hint that
> > > allows
> > > > >> writers to error out once they see a NaN or INF in the input,
> while
> > > they
> > > > >> will still happily write a bogus 0 vector. This seems to be of
> zero
> > > > >> benefit.
> > > > >>
> > > > >> Anecdotally, we currently handle petabytes of vectors in
> production
> > in
> > > > >> Data360. We simply store them in a List of Float, with an
> optimized
> > > > R-Level
> > > > >> reader implementation that rapidly checks that all vectors are the
> > > same
> > > > >> length. This solution works just fine. The overhead of R-level
> > > decoding
> > > > is
> > > > >> barely measurable. The check is super fast and R-levels are tiny
> in
> > > > >> comparison to the payload vectors, so also the added I/O is
> > > negligible.
> > > > >> This is also why I'm so firm in my sentiment: We are running a
> > > > production
> > > > >> grade system processing trillions of vectors right now and see no
> > big
> > > > >> issues with the types that Parquet currently provides.
> > > > >>
> > > > >> As I don't just want to be a naysayer and point out problems with
> > the
> > > > >> current design, here are the directions that I would propose
> indeed:
> > > > >>
> > > > >>   - a general fixed size list type. Don't do shortcuts just
> because
> > > the
> > > > >>   discussion is too tedious.
> > > > >>   - a vector type that indeed carries enough information to be
> > really
> > > > >>   useful for vector search by itself. (e.g., model name / version,
> > > > >> optional
> > > > >>   guarantees for no 0 vector / normalized vector length). With
> such
> > a
> > > > >> type,
> > > > >>   you could really do well defined vector searches even if the
> > writing
> > > > >> engine
> > > > >>   is completely different to the reading one.
> > > > >>   - Standardize quantizations. This will be a real game changer
> for
> > > > >>   performance.
> > > > >>
> > > > >> Cheers,
> > > > >> Jan
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >>> Am Mi., 14. Okt. 2026 um 03:36 Uhr schrieb Gang Wu <
> > ustcwg@gmail.com
> > > >:
> > > > >>>
> > > > >>> I think separating them into FINITE_VECTOR and FIXED_LENGTH_LIST
> > is a
> > > > >> good
> > > > >>> idea, though I think renaming FINITE_VECTOR to VECTOR looks more
> > > > natural.
> > > > >>> Should we move forward with proposing FINITE_VECTOR/VECTOR first?
> > > > >>>
> > > > >>> On Tue, Oct 13, 2026 at 6:26 PM Alkis Evlogimenos via dev <
> > > > >>> dev@parquet.apache.org> wrote:
> > > > >>>
> > > > >>>> I suggest we go with VECTOR for the finite vector to keep the
> name
> > > the
> > > > >>> same
> > > > >>>> as other engines/type systems. If we want a non-finite one in
> the
> > > > >> future
> > > > >>> we
> > > > >>>> can name that something else or we add fixed size list that will
> > > serve
> > > > >>> the
> > > > >>>> same purpose without a special name.
> > > > >>>>
> > > > >>>>
> > > > >>>> On Thu, Oct 8, 2026 at 10:20 PM Ryan Blue <bl...@apache.org>
> > wrote:
> > > > >>>>
> > > > >>>>> A logical type would:
> > > > >>>>> - Describe data for users and readers to understand what the
> file
> > > > >>>> contains
> > > > >>>>> - Signal that repetition and definition levels will reflect a
> > fixed
> > > > >>>> number
> > > > >>>>> of elements, which implementations can use to optimize decoding
> > > > >>>>> - Guarantee that there are no null elements (because the type
> > > cannot
> > > > >> be
> > > > >>>>> applied to a list with optional elements)
> > > > >>>>> - [FINITE_VECTOR] Guarantee that no elements are NaN or
> Infinity
> > > > >> values
> > > > >>>> so
> > > > >>>>> that the vectors have a finite length and are suitable for
> > distance
> > > > >>>>> calculations
> > > > >>>>>
> > > > >>>>> The finite value restriction is a restriction on writers to
> > prevent
> > > > >> bad
> > > > >>>>> data from being written.
> > > > >>>>>
> > > > >>>>> Ignoring logical types that are not understood is a rule for
> > > > >> readers. A
> > > > >>>>> reader can handle vectors and pass it on to another layer. For
> > > > >>> instance,
> > > > >>>> a
> > > > >>>>> Parquet reader that doesn't support geometry passes values as
> > > > >> `binary`.
> > > > >>>>> That binary value can be passed to a geometry UDF, even if the
> > > > >>> underlying
> > > > >>>>> engine and Parquet reader doesn't have support.
> > > > >>>>>
> > > > >>>>> A tool that doesn't understand FINITE_VECTOR can similarly
> pass a
> > > > >> list
> > > > >>> of
> > > > >>>>> floats along to a UDF. But it is still important to the user
> that
> > > bad
> > > > >>>>> values are not present in the data. In your example, bad values
> > are
> > > > >>>> vectors
> > > > >>>>> that cause the centroid calculation to fail, like vectors with
> > > > >> infinite
> > > > >>>>> length.
> > > > >>>>>
> > > > >>>>> Ryan
> > > > >>>>>
> > > > >>>>> On Thu, Oct 8, 2026 at 12:54 PM Will Edwards via dev <
> > > > >>>>> dev@parquet.apache.org>
> > > > >>>>> wrote:
> > > > >>>>>
> > > > >>>>>> Could we clarify the use cases for a VECTOR annotation on
> > > > >>> LIST<FLOAT>?
> > > > >>>>>> Specifically, what does it make possible that's impossible
> > today,
> > > > >> and
> > > > >>>>> what
> > > > >>>>>> does it make easier that's currently hard?
> > > > >>>>>>
> > > > >>>>>> I'm struggling to find a strong answer. I'd like every engine
> to
> > > > >>> offer
> > > > >>>>>> vector functions on lists of floats without needing a new
> > logical
> > > > >>> type.
> > > > >>>>>>
> > > > >>>>>> To be clear, I'm still enthusiastic about a vector type that
> > > > >>> describes
> > > > >>>>> how
> > > > >>>>>> to unpack and scale composite representations (quantised codes
> > > plus
> > > > >>>>> scales,
> > > > >>>>>> codebooks, etc.), in the direction ML is going. That adds
> > > > >> information
> > > > >>>>>> engines don't have. A label on LIST<FLOAT> doesn't: engines
> > > already
> > > > >>>>>> understand the physical type, and many already offer vector
> > > > >> functions
> > > > >>>> on
> > > > >>>>>> it. I don't see a performance benefit, and NaN/null/Inf checks
> > are
> > > > >>>> cheap
> > > > >>>>>> enough that engines should do them anyway. The label also
> > doesn't
> > > > >> say
> > > > >>>>> what
> > > > >>>>>> you can do with the column. Knowing that a column holds
> > embeddings
> > > > >>>>> doesn't
> > > > >>>>>> say how they can be used, just as zipcode and age are both
> > > integers
> > > > >>> but
> > > > >>>>>> multiplying them is meaningless.
> > > > >>>>>>
> > > > >>>>>> My concern is the opposite effect. We're establishing that
> > engines
> > > > >>>> should
> > > > >>>>>> ignore logical types they don't understand. So a tool that
> > doesn't
> > > > >>> know
> > > > >>>>>> VECTOR will read the column as LIST<FLOAT>, let the user
> compute
> > > > >>>>> centroids
> > > > >>>>>> (say), and write plain LIST<FLOAT> back out. If other engines
> > > > >>> restrict
> > > > >>>>>> embedding functions to VECTOR-annotated columns, that file
> stops
> > > > >>>> working
> > > > >>>>>> with them, through no fault of the user. An annotation that
> > > doesn't
> > > > >>>>> change
> > > > >>>>>> what the data is shouldn't be able to make it unusable.
> > > > >>>>>>
> > > > >>>>>> Am I missing a use case?
> > > > >>>>>>
> > > > >>>>>> Best, Will
> > > > >>>>>>
> > > > >>>>>> On Thu, 8 Oct 2026 at 19:53, Steven Wu <st...@gmail.com>
> > > > >> wrote:
> > > > >>>>>>
> > > > >>>>>>> I also think a FINITE_VECTOR type brings a lot of values to
> > many
> > > > >>> use
> > > > >>>>>> cases.
> > > > >>>>>>> It is would be great to make progress on the FINITE_VECTOR,
> and
> > > > >> not
> > > > >>>> to
> > > > >>>>>> fit
> > > > >>>>>>> every use cases into the FIXED_LENGTH_LIST.
> > > > >>>>>>>
> > > > >>>>>>> On Mon, Oct 5, 2026 at 1:26 PM Ryan Blue <bl...@apache.org>
> > > > >> wrote:
> > > > >>>>>>>
> > > > >>>>>>>> Going through this thread again, it looks like there are a
> few
> > > > >>>> things
> > > > >>>>>>> that
> > > > >>>>>>>> people generally agree on:
> > > > >>>>>>>>
> > > > >>>>>>>> - In the short term, we want to introduce a logical type or
> > > > >>> types;
> > > > >>>> a
> > > > >>>>>>> better
> > > > >>>>>>>> physical representation can be done later. We can focus on
> > > > >>> logical
> > > > >>>>>> types.
> > > > >>>>>>>> - There is precedent in logical types for restricting
> values,
> > > > >>> like
> > > > >>>>> UTF8
> > > > >>>>>>>> strings to contain only UTF-8 codepoints. I think we also
> > agree
> > > > >>>> that
> > > > >>>>> we
> > > > >>>>>>>> could restrict vector elements to non-nested or primitive
> > > > >> types.
> > > > >>>>>>>> - There are many examples of databases that restrict their
> > > > >> vector
> > > > >>>>> types
> > > > >>>>>>> to
> > > > >>>>>>>> non-null, finite elements.
> > > > >>>>>>>> - Some people want to have write-time guarantees that
> vectors
> > > > >>> have
> > > > >>>>> only
> > > > >>>>>>>> non-null, finite elements
> > > > >>>>>>>> - Some people want to extend this work to include tensors,
> > > > >>> shorter
> > > > >>>>>>> floating
> > > > >>>>>>>> point representations, variable lengths, NaN or Infinity
> > > > >>> elements,
> > > > >>>>> etc.
> > > > >>>>>>>>
> > > > >>>>>>>> It sounds like we are nearly there, but have two halves of
> the
> > > > >>>>>> community
> > > > >>>>>>>> that care about different things. That just means we should
> > > > >> have
> > > > >>>>>> multiple
> > > > >>>>>>>> types. It's entirely reasonable to want a type that allows
> NaN
> > > > >>>>> values,
> > > > >>>>>>> and
> > > > >>>>>>>> it's also reasonable for those of us building databases,
> table
> > > > >>>>>> standards,
> > > > >>>>>>>> and vector indexes to care about guarantees that elements
> are
> > > > >>>> finite.
> > > > >>>>>>>> There's more than enough examples to show that this is a
> > > > >>> legitimate
> > > > >>>>> use
> > > > >>>>>>>> case.
> > > > >>>>>>>>
> > > > >>>>>>>> Since I don't see the concerns of the different use cases
> > > > >>> changing,
> > > > >>>>> it
> > > > >>>>>>>> looks like we need to find a compromise to move forward. I
> > > > >>> propose
> > > > >>>>> that
> > > > >>>>>>> we
> > > > >>>>>>>> introduce a FINITE_VECTOR type and separately a
> > > > >> FIXED_LENGTH_LIST
> > > > >>>> or
> > > > >>>>>>>> another VECTOR type that allows infinite and NaN values.
> > > > >>>>>>>>
> > > > >>>>>>>> Can we agree on a compromise? Is there a strong reason to
> try
> > > > >> to
> > > > >>>> fit
> > > > >>>>>>> these
> > > > >>>>>>>> use cases into a single type? What if that means not making
> > > > >>>> progress?
> > > > >>>>>>>>
> > > > >>>>>>>> Ryan
> > > > >>>>>>>>
> > > > >>>>>>>> On Mon, Oct 5, 2026 at 12:15 PM Philipp Fischbeck via dev <
> > > > >>>>>>>> dev@parquet.apache.org> wrote:
> > > > >>>>>>>>
> > > > >>>>>>>>> Thinking about this more, I am also fine with pulling out
> the
> > > > >>>>>> “NaN/Inf”
> > > > >>>>>>>>> requirement. Either into a type boolean (my preference), or
> > > > >>>> putting
> > > > >>>>>> no
> > > > >>>>>>>>> restrictions on the Parquet level, allowing all IEEE 754
> > > > >> float
> > > > >>>>>> numbers.
> > > > >>>>>>>> If
> > > > >>>>>>>>> some top-level application wants to put additional
> > > > >> constraints
> > > > >>> on
> > > > >>>>> the
> > > > >>>>>>>> data
> > > > >>>>>>>>> type, like Iceberg not allowing NaN/infinity, it can still
> do
> > > > >>> so
> > > > >>>>> one
> > > > >>>>>>>> layer
> > > > >>>>>>>>> above. It seems like this was the biggest point of
> discussion
> > > > >>>>>>> remaining.
> > > > >>>>>>>>> All these options work for me, and we should move forward
> > > > >> with
> > > > >>>> this
> > > > >>>>>>>>> proposal.
> > > > >>>>>>>>>
> > > > >>>>>>>>> Concerning additional element types like bf16, they can be
> > > > >>>>> introduced
> > > > >>>>>>> as
> > > > >>>>>>>>> Parquet types (at least in the current LIST-based spec
> > > > >>> proposal).
> > > > >>>>>>>>>
> > > > >>>>>>>>> Concerning Will’s suggestion around representing VECTOR as
> a
> > > > >>>>> variable
> > > > >>>>>>>>> length list with a MIN/MAX_LENGTH metadata restriction: I
> > > > >> think
> > > > >>>>> that
> > > > >>>>>> a
> > > > >>>>>>>>> dedicated logical VECTOR type still provides enough
> benefits
> > > > >> by
> > > > >>>>>>>> describing
> > > > >>>>>>>>> the semantics of the data, and allowing to capture future
> > > > >>> changes
> > > > >>>>>>> around
> > > > >>>>>>>>> statistics. In that case, the length is better captured as
> an
> > > > >>>> exact
> > > > >>>>>>>> length
> > > > >>>>>>>>> for this type specifically. I’m also not sure about other
> use
> > > > >>>> cases
> > > > >>>>>> for
> > > > >>>>>>>>> MIN_LENGTH, so introducing a new type constraint system
> just
> > > > >>> for
> > > > >>>>> the
> > > > >>>>>>>>> MIN_LENGTH=MAX_LENGTH case for vectors doesn’t have enough
> > > > >>> value
> > > > >>>> in
> > > > >>>>>> my
> > > > >>>>>>>>> view.
> > > > >>>>>>>>>
> > > > >>>>>>>>> Best,
> > > > >>>>>>>>> Philipp
> > > > >>>>>>>>>
> > > > >>>>>>>>>
> > > > >>>>>>>>> On Thu, Oct 1, 2026 at 9:00 AM Will Edwards via dev <
> > > > >>>>>>>>> dev@parquet.apache.org>
> > > > >>>>>>>>> wrote:
> > > > >>>>>>>>>
> > > > >>>>>>>>>> +1 Sebastian, a vector type suggests usage beyond
> > > > >> normalized
> > > > >>>>>>> embeddings
> > > > >>>>>>>>> and
> > > > >>>>>>>>>> engines really aren't going to be able to elide all the
> > > > >>> checks
> > > > >>>>>>> anyway.
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> Antoine, you suggested we explore metadata solutions too
> :)
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> Today, a database supporting CHAR(N) or VARCHAR(n) can't
> > > > >>>> express
> > > > >>>>>> this
> > > > >>>>>>>> in
> > > > >>>>>>>>>> Parquet.  Engines can write opaque blobs they understand
> in
> > > > >>> the
> > > > >>>>>>>> metadata
> > > > >>>>>>>>>> (e.g. Arrow does) so the extra type info round-trips, but
> > > > >>> this
> > > > >>>>>> isn't
> > > > >>>>>>>>>> standardized, portable, nor interoperable.
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> So perhaps standardizing constraint metadata or new Thrift
> > > > >>>> fields
> > > > >>>>>> (my
> > > > >>>>>>>>> pref)
> > > > >>>>>>>>>> can generically support VECTOR(N) whilst also being more
> > > > >>>> broadly
> > > > >>>>>>> useful
> > > > >>>>>>>>> for
> > > > >>>>>>>>>> other use cases and data types?
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> Imagine metadata or Thrift fields to tag the MIN_LENGTH
> and
> > > > >>>>>>> MAX_LENGTH
> > > > >>>>>>>>> for
> > > > >>>>>>>>>> Parquet's variable-length physcial types.  These would
> > > > >> count
> > > > >>>>>> logical
> > > > >>>>>>>>> units
> > > > >>>>>>>>>> (e.g. UTF8 code points for STRING) rather than bytes.
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> Then one engine's VECTOR(N) can be written to Parquet as
> > > > >>>> LIST(N)
> > > > >>>>>> with
> > > > >>>>>>>>>> MIN_LENGTH = MAX_LENGTH = N and then readers - even
> > > > >> different
> > > > >>>>>>> engines -
> > > > >>>>>>>>> can
> > > > >>>>>>>>>> pick up these constraints if they support doing so.  We
> > > > >> don't
> > > > >>>>>>> actually
> > > > >>>>>>>>> need
> > > > >>>>>>>>>> a logical type to express a VECTOR then.
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> This also allows standard SQL's CHAR(N) and VARCHAR(N) to
> > > > >> be
> > > > >>>>>>> expressed
> > > > >>>>>>>> on
> > > > >>>>>>>>>> UTF8 columns etc.
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> This also seems generally useful in the future as we
> > > > >> approach
> > > > >>>>>>>> expressing
> > > > >>>>>>>>>> other tensor data shapes and types.
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> This won't change the mechanics: the storage type doesn't
> > > > >>>> enforce
> > > > >>>>>>> these
> > > > >>>>>>>>>> constraints and engines must check them. It's previously
> > > > >>>>>> established
> > > > >>>>>>>> that
> > > > >>>>>>>>>> performance is just a software problem and there is no
> > > > >>>>> fundamental
> > > > >>>>>>>>>> performance advantage to the fixed size types in
> > > > >> Dremel-like
> > > > >>>>>> formats
> > > > >>>>>>>> like
> > > > >>>>>>>>>> Parquet.
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> Best,
> > > > >>>>>>>>>> Will
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> On Wed, 30 Sept 2026 at 16:08, Sebastian Baunsgaard <
> > > > >>>>>>>>> baunsgaard@apache.org
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>> wrote:
> > > > >>>>>>>>>>
> > > > >>>>>>>>>>> After listening in on the last meeting Wednesday, I have
> > > > >>> been
> > > > >>>>>>>> thinking
> > > > >>>>>>>>>>> about this quite a bit, and thought I would share my 2
> > > > >>> cents.
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> I suggest not requiring VECTOR elements to be finite, as
> > > > >>>>> proposed
> > > > >>>>>>> in
> > > > >>>>>>>>> the
> > > > >>>>>>>>>> PR
> > > > >>>>>>>>>>> <https://github.com/apache/parquet-format/pull/624>.
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> While Divjot and others note the specialization for
> > > > >> UTF-8,
> > > > >>>>>> Parquet
> > > > >>>>>>>>>> already
> > > > >>>>>>>>>>> defines DOUBLE and FLOAT as IEEE encodings
> > > > >>>>>>>>>>> <
> > > > >>>>>>>>
> > > > >>>>
> > > https://github.com/apache/parquet-format/blob/master/README.md#types
> > > > >>>>>>>>>> .
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> Making NaN or +/-Inf illegal inside VECTOR would be a new
> > > > >>>>>>> restriction
> > > > >>>>>>>>> on
> > > > >>>>>>>>>>> those same underlying physical types. Scalar FLOAT/DOUBLE
> > > > >>>>> columns
> > > > >>>>>>>> have
> > > > >>>>>>>>>>> never had such a rule.
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> It would also not match how the systems that actually
> > > > >>> produce
> > > > >>>>>> these
> > > > >>>>>>>>>> vectors
> > > > >>>>>>>>>>> treat floats. Most of them use IEEE floats, e.g. Spark,
> > > > >>>> DuckDB,
> > > > >>>>>>>> NumPy,
> > > > >>>>>>>>>>> PyTorch, TensorFlow, or lower level CUDA, oneMKL, and
> > > > >>>> OpenBLAS.
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> pgvector's "all elements must be finite" rule is an
> > > > >> insert
> > > > >>>>>> policy.
> > > > >>>>>>>>> While
> > > > >>>>>>>>>> I
> > > > >>>>>>>>>>> think Parquet as a general-purpose format should allow
> > > > >> them
> > > > >>>> out
> > > > >>>>>> of
> > > > >>>>>>>> the
> > > > >>>>>>>>>> box.
> > > > >>>>>>>>>>> It is a user system decision to disallow writing these
> > > > >>> values
> > > > >>>>> or
> > > > >>>>>> to
> > > > >>>>>>>>> throw
> > > > >>>>>>>>>>> errors when encountering them.
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> A type-level ban also does not keep Inf/NaN out of vector
> > > > >>>> math.
> > > > >>>>>>>> Finite
> > > > >>>>>>>>>>> coordinates still overflow. For instance: magnitude =
> > > > >>>>>> ||[FLOAT_MAX,
> > > > >>>>>>>>>>> FLOAT_MAX]|| where every coordinate is finite overflows
> > > > >> to
> > > > >>>>> +Inf.
> > > > >>>>>>>>> PyTorch
> > > > >>>>>>>>>>> documents the same overflow: tensor([1e20, 1e20]).norm()
> > > > >> is
> > > > >>>> inf
> > > > >>>>>> in
> > > > >>>>>>>>> fp32.
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> A generated MAGNITUDE(vec) column would then be Inf while
> > > > >>> the
> > > > >>>>>>> VECTOR
> > > > >>>>>>>>>> itself
> > > > >>>>>>>>>>> was required to be finite. The ban only forbids storing
> > > > >>>>> specials
> > > > >>>>>> in
> > > > >>>>>>>> the
> > > > >>>>>>>>>>> coordinates, and does not prevent even simple downstream
> > > > >>>>>>> computations
> > > > >>>>>>>>>> from
> > > > >>>>>>>>>>> producing them.
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> I suggest keeping VECTOR as the type and add an optional
> > > > >>> bool
> > > > >>>>>>>>>>> VectorType.finite (default false). Engines that want
> > > > >>>>>> pgvector-like
> > > > >>>>>>>>>>> rejection can require the flag = true. Engines that use
> > > > >>>> NumPy /
> > > > >>>>>>>>> PyTorch /
> > > > >>>>>>>>>>> Spark arrays can keep representable IEEE values. For the
> > > > >>> same
> > > > >>>>>>>> reason, I
> > > > >>>>>>>>>> do
> > > > >>>>>>>>>>> not favor adding yet another type called FINITE_VECTOR.
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> Best,
> > > > >>>>>>>>>>> Sebastian
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> On Tue, Sep 29, 2026 at 2:03 PM Divjot Arora via dev <
> > > > >>>>>>>>>>> dev@parquet.apache.org>
> > > > >>>>>>>>>>> wrote:
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>>> I agree with moving forward with the proposal and
> > > > >>> renaming
> > > > >>>>>> VECTOR
> > > > >>>>>>>> to
> > > > >>>>>>>>>>>> FINITE_VECTOR or something
> > > > >>>>>>>>>>>> similar if others feel that better communicates its
> > > > >>>> intent. I
> > > > >>>>>> see
> > > > >>>>>>>>> value
> > > > >>>>>>>>>>> in
> > > > >>>>>>>>>>>> having a vector type that guarantees
> > > > >>>>>>>>>>>> its values are mathematically-correct vectors for a few
> > > > >>>>>> reasons:
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> 1. As others have mentioned, this is similar to the
> > > > >>> STRING
> > > > >>>>>>> logical
> > > > >>>>>>>>> type
> > > > >>>>>>>>>>>> mandating UTF-8.
> > > > >>>>>>>>>>>> 2. It is not possible to rely on statistics for
> > > > >> detecting
> > > > >>>> NaN
> > > > >>>>>> or
> > > > >>>>>>>>> +-inf
> > > > >>>>>>>>>>>> values as all statistics are optional.
> > > > >>>>>>>>>>>> To my knowledge, there is no precedent for requiring
> > > > >> that
> > > > >>>>>> writers
> > > > >>>>>>>>> emit
> > > > >>>>>>>>>>>> stats for specific logical types.
> > > > >>>>>>>>>>>> 3. Without statistics, NaN/inf values could flow
> > > > >> through
> > > > >>>> the
> > > > >>>>>>> engine
> > > > >>>>>>>>> and
> > > > >>>>>>>>>>>> block them for specific functionality.
> > > > >>>>>>>>>>>> This seems to be Lance's behavior. However, I don't
> > > > >> think
> > > > >>>>> this
> > > > >>>>>> is
> > > > >>>>>>>> as
> > > > >>>>>>>>>> easy
> > > > >>>>>>>>>>>> as it sounds and Lance has run
> > > > >>>>>>>>>>>> into several bugs due to this ([1] [2] for examples).
> > > > >>> Their
> > > > >>>>>>> Python
> > > > >>>>>>>>> [3]
> > > > >>>>>>>>>>> and
> > > > >>>>>>>>>>>> Rust [4] APIs default to erroring on
> > > > >>>>>>>>>>>> vectors that contain NaN values.
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> I agree that Parquet is a general-purpose format, but I
> > > > >>>> don't
> > > > >>>>>>> think
> > > > >>>>>>>>>> this
> > > > >>>>>>>>>>>> implies the lack of domain-specific types.
> > > > >>>>>>>>>>>> Instead, this broadens the use cases where Parquet
> > > > >> would
> > > > >>> be
> > > > >>>>>>>>> considered
> > > > >>>>>>>>>> a
> > > > >>>>>>>>>>>> good fit. I also fully agree with
> > > > >>>>>>>>>>>> continuing work on a general fixed-size list type if we
> > > > >>>>> believe
> > > > >>>>>>>> there
> > > > >>>>>>>>>> are
> > > > >>>>>>>>>>>> non-vector use cases that would benefit
> > > > >>>>>>>>>>>> from such guarantees at the storage layer.
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> [1] https://github.com/lance-format/lance/pull/3648
> > > > >>>>>>>>>>>> [2] https://github.com/lance-format/lance/issues/8607
> > > > >>> (see
> > > > >>>>>>> point 2
> > > > >>>>>>>>> in
> > > > >>>>>>>>>>>> "root
> > > > >>>>>>>>>>>> cause")
> > > > >>>>>>>>>>>> [3]
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>
> > > > >>>>>>>>>
> > > > >>>>>>>>
> > > > >>>>>>>
> > > > >>>>>>
> > > > >>>>>
> > > > >>>>
> > > > >>>
> > > > >>
> > > >
> > >
> >
> https://github.com/lancedb/lancedb/blob/0327618a62483feb6dca2336549845e5887cd48f/python/python/lancedb/table.py#L4807-L4911
> > > > >>>>>>>>>>>> [4]
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>
> > > > >>>>>>>>>
> > > > >>>>>>>>
> > > > >>>>>>>
> > > > >>>>>>
> > > > >>>>>
> > > > >>>>
> > > > >>>
> > > > >>
> > > >
> > >
> >
> https://github.com/lancedb/lancedb/blob/0327618a62483feb6dca2336549845e5887cd48f/rust/lancedb/src/table/add_data.rs#L105-L110
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> On Mon, Sep 28, 2026 at 6:19 PM Philipp Fischbeck via
> > > > >>> dev <
> > > > >>>>>>>>>>>> dev@parquet.apache.org> wrote:
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>>> I also favor moving forward with the proposal as is.
> > > > >> I
> > > > >>>> see
> > > > >>>>>>> strong
> > > > >>>>>>>>>> value
> > > > >>>>>>>>>>>> and
> > > > >>>>>>>>>>>>> use cases for the numeric vector type, and I agree
> > > > >> with
> > > > >>>> the
> > > > >>>>>>>>>>> restrictions
> > > > >>>>>>>>>>>>> around NaN/Inf values and element types in general.
> > > > >>>>> Precedent
> > > > >>>>>>>>> exists
> > > > >>>>>>>>>> in
> > > > >>>>>>>>>>>>> other types that restrict which physical type values
> > > > >>> are
> > > > >>>>>>>> logically
> > > > >>>>>>>>>>> valid
> > > > >>>>>>>>>>>>> (e.g., the geospatial types needing to be
> > > > >> WKB-encoded).
> > > > >>>> A
> > > > >>>>>>>> general
> > > > >>>>>>>>>>>>> fixed-size list also has value, but I think it makes
> > > > >>>> sense
> > > > >>>>> to
> > > > >>>>>>>>>> separate
> > > > >>>>>>>>>>>> the
> > > > >>>>>>>>>>>>> two types. The semantics are relevant, and I can see
> > > > >>> how
> > > > >>>>> the
> > > > >>>>>>> two
> > > > >>>>>>>>>> types
> > > > >>>>>>>>>>>>> might diverge further in the future regarding
> > > > >>> statistics
> > > > >>>> or
> > > > >>>>>>>>>> encodings.
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> I would be okay with renaming the vector type, but it
> > > > >>>> seems
> > > > >>>>>>> like
> > > > >>>>>>>>> the
> > > > >>>>>>>>>>>> VECTOR
> > > > >>>>>>>>>>>>> name is already used across other engines mentioned
> > > > >> in
> > > > >>>> this
> > > > >>>>>>>> thread.
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> Best,
> > > > >>>>>>>>>>>>> Philipp
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> On Sat, Sep 26, 2026 at 4:37 PM Gang Wu <
> > > > >>>> ustcwg@gmail.com>
> > > > >>>>>>>> wrote:
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> To separate orthogonal concerns, I feel we should
> > > > >>> treat
> > > > >>>>> the
> > > > >>>>>>>>> general
> > > > >>>>>>>>>>>>>> FixedSizeList as the underlying physical layout,
> > > > >> and
> > > > >>>>>> separate
> > > > >>>>>>>> it
> > > > >>>>>>>>>> from
> > > > >>>>>>>>>>>> the
> > > > >>>>>>>>>>>>>> discussion of a specialized VECTOR (or broader ML
> > > > >>>> tensor)
> > > > >>>>>>>> logical
> > > > >>>>>>>>>>> type.
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> Currently, the immediate need is to define a
> > > > >>>> specialized
> > > > >>>>>>> VECTOR
> > > > >>>>>>>>>> type
> > > > >>>>>>>>>>> so
> > > > >>>>>>>>>>>>>> that different engines and formats have a common
> > > > >>>> semantic
> > > > >>>>>>>>> contract
> > > > >>>>>>>>>>> for
> > > > >>>>>>>>>>>>>> interoperability.
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> If the main point of contention is just the
> > > > >>>> NaN/Inf/null
> > > > >>>>>>>>>> limitations,
> > > > >>>>>>>>>>>>> maybe
> > > > >>>>>>>>>>>>>> we can resolve this by adding a qualification to
> > > > >> the
> > > > >>>> name
> > > > >>>>>>> (like
> > > > >>>>>>>>>>> Andrew
> > > > >>>>>>>>>>>>>> suggested, e.g., VECTOR_EMBEDDING or FINITE_VECTOR)
> > > > >>> to
> > > > >>>>>> avoid
> > > > >>>>>>>>>>> confusion
> > > > >>>>>>>>>>>>> with
> > > > >>>>>>>>>>>>>> broader ML use cases. Alternatively, we could
> > > > >>>> introduce a
> > > > >>>>>>>> boolean
> > > > >>>>>>>>>>> flag
> > > > >>>>>>>>>>>>>> (e.g., optional bool VectorType.finite) as proposed
> > > > >>>>>> earlier.
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> On Fri, Sep 25, 2026 at 3:52 AM Andrew Lamb <
> > > > >>>>>>>>>> andrewlamb11@gmail.com>
> > > > >>>>>>>>>>>>>> wrote:
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> I am a little confused about the points of
> > > > >>>> disagreement
> > > > >>>>>>> here.
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> Is it that the name `VECTOR` is too general for
> > > > >> the
> > > > >>>>>>> specific
> > > > >>>>>>>>> use
> > > > >>>>>>>>>>> case
> > > > >>>>>>>>>>>>> of
> > > > >>>>>>>>>>>>>>> fixed length arrays with non-finite values used
> > > > >> for
> > > > >>>>>> vector
> > > > >>>>>>>>>>>> embeddings?
> > > > >>>>>>>>>>>>>> If
> > > > >>>>>>>>>>>>>>> so, maybe we could add a qualification to the
> > > > >> name
> > > > >>>>>>>>>> (FINITE_VECTOR?
> > > > >>>>>>>>>>>>>>> FEATURE_VECTOR? VECTOR_EMBEDDING?)
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> It seems like there is precedent for defining
> > > > >>> logical
> > > > >>>>>> types
> > > > >>>>>>>>> that
> > > > >>>>>>>>>>>>> specify
> > > > >>>>>>>>>>>>>>> certain physical types and restrict their
> > > > >> contents
> > > > >>> to
> > > > >>>>>> help
> > > > >>>>>>>>>> readers
> > > > >>>>>>>>>>>>>>> interpret meaning, for example string[1] or JSON;
> > > > >>>>>> Defining
> > > > >>>>>>>> one
> > > > >>>>>>>>>> for
> > > > >>>>>>>>>>>>>> vector
> > > > >>>>>>>>>>>>>>> embeddings seems reasonable enough to me.
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> Andrew
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> [1]:
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>
> > > > >>>>>>>>>
> > > > >>>>>>>>
> > > > >>>>>>>
> > > > >>>>>>
> > > > >>>>>
> > > > >>>>
> > > > >>>
> > > > >>
> > > >
> > >
> >
> https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#string
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> On Thu, Sep 24, 2026 at 3:34 PM Daniel Weeks <
> > > > >>>>>>>>> dweeks@apache.org>
> > > > >>>>>>>>>>>>> wrote:
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>> Well, Parquet is not a single-purpose format.
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>> Vector stores can choose whatever policy fits
> > > > >>>> their
> > > > >>>>>>>> usage,
> > > > >>>>>>>>>> but
> > > > >>>>>>>>>>>>> we're
> > > > >>>>>>>>>>>>>>>>> talking about a general-purpose file format
> > > > >>> that
> > > > >>>>> aims
> > > > >>>>>>> to
> > > > >>>>>>>> be
> > > > >>>>>>>>>>>> broadly
> > > > >>>>>>>>>>>>>>>>> applicable.
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>> I agree that the format is intended to be
> > > > >> general
> > > > >>>>>>> purpose,
> > > > >>>>>>>>> but
> > > > >>>>>>>>>>>> types
> > > > >>>>>>>>>>>>>>> within
> > > > >>>>>>>>>>>>>>>> the format define specific representation and
> > > > >>>>> semantics
> > > > >>>>>>> to
> > > > >>>>>>>>>>> achieve
> > > > >>>>>>>>>>>>> that
> > > > >>>>>>>>>>>>>>>> goal.
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>> "Vector" has a more general sense than ML and
> > > > >>>> it's
> > > > >>>>>>> quite
> > > > >>>>>>>>>>> trivial
> > > > >>>>>>>>>>>> to
> > > > >>>>>>>>>>>>>>>>> produce vectors with non-finite values using
> > > > >>> e.g.
> > > > >>>>>> NumPy
> > > > >>>>>>>> or
> > > > >>>>>>>>>>> Pandas
> > > > >>>>>>>>>>>>> (or
> > > > >>>>>>>>>>>>>>>>> any other piece of dataframe or scientific
> > > > >>>>> computing
> > > > >>>>>>>>>> software).
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>> The justification for restricting non-finite
> > > > >>>> values
> > > > >>>>> is
> > > > >>>>>>>>>> directly
> > > > >>>>>>>>>>>>>> related
> > > > >>>>>>>>>>>>>>> to
> > > > >>>>>>>>>>>>>>>> the semantic meaning of a vector.  Vectors have
> > > > >>>>>> specific
> > > > >>>>>>>>>>>>>> characteristics
> > > > >>>>>>>>>>>>>>>> (dimension, magnitude, direction, etc).
> > > > >>> Non-finite
> > > > >>>>>>> values
> > > > >>>>>>>>> are
> > > > >>>>>>>>>>>>> invalid
> > > > >>>>>>>>>>>>>>> and
> > > > >>>>>>>>>>>>>>>> break those characteristics.  I'd liken it to
> > > > >>>> storing
> > > > >>>>>>>>> non-utf8
> > > > >>>>>>>>>>>> values
> > > > >>>>>>>>>>>>>> in
> > > > >>>>>>>>>>>>>>> a
> > > > >>>>>>>>>>>>>>>> string and still calling it a string.
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>> That also differentiates a vector from a fixed
> > > > >>>> length
> > > > >>>>>>> list,
> > > > >>>>>>>>>> which
> > > > >>>>>>>>>>>> is
> > > > >>>>>>>>>>>>>> more
> > > > >>>>>>>>>>>>>>>> general and does not carry the same semantic
> > > > >>>> meaning.
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>> I also don't think we need to bundle all
> > > > >>> similarly
> > > > >>>>>> shaped
> > > > >>>>>>>> use
> > > > >>>>>>>>>>> cases
> > > > >>>>>>>>>>>>>> into
> > > > >>>>>>>>>>>>>>>> the same solution (e.g. flxed size list).
> > > > >> Vector
> > > > >>>> is
> > > > >>>>> a
> > > > >>>>>>> well
> > > > >>>>>>>>>>>>> established
> > > > >>>>>>>>>>>>>>>> data type at this point and aligns with the
> > > > >>> general
> > > > >>>>>>> purpose
> > > > >>>>>>>>>>> nature
> > > > >>>>>>>>>>>> of
> > > > >>>>>>>>>>>>>> the
> > > > >>>>>>>>>>>>>>>> format (many other formats and databases
> > > > >> natively
> > > > >>>>>> support
> > > > >>>>>>>>>> this).
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>> -Dan
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>> On Thu, Sep 24, 2026 at 11:55 AM Antoine
> > > > >> Pitrou <
> > > > >>>>>>>>>>>> antoine@python.org>
> > > > >>>>>>>>>>>>>>>> wrote:
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>> Le 24/09/2026 à 20:11, Russell Spitzer a
> > > > >> écrit
> > > > >>> :
> > > > >>>>>>>>>>>>>>>>>> I think something I haven't seen yet is the
> > > > >>>> clear
> > > > >>>>>>>>> use-case
> > > > >>>>>>>>>>> for
> > > > >>>>>>>>>>>>>>>> non-finite
> > > > >>>>>>>>>>>>>>>>>> values within a vector type. There has
> > > > >> been a
> > > > >>>> lot
> > > > >>>>>> of
> > > > >>>>>>>>>>>> theoretical
> > > > >>>>>>>>>>>>>>>>> discussion
> > > > >>>>>>>>>>>>>>>>>> here about use cases but what toolings
> > > > >>>> currently
> > > > >>>>>>>> produce
> > > > >>>>>>>>>>> these
> > > > >>>>>>>>>>>>>>> vectors
> > > > >>>>>>>>>>>>>>>>> and
> > > > >>>>>>>>>>>>>>>>>> how do they function in downstream engines?
> > > > >>> I'd
> > > > >>>>>> like
> > > > >>>>>>> to
> > > > >>>>>>>>>> limit
> > > > >>>>>>>>>>>>>>>> introducing
> > > > >>>>>>>>>>>>>>>>>> types that have a wide range of values
> > > > >> which
> > > > >>>> are
> > > > >>>>>>> simply
> > > > >>>>>>>>> are
> > > > >>>>>>>>>>>>> failure
> > > > >>>>>>>>>>>>>>>> modes
> > > > >>>>>>>>>>>>>>>>>> of different flavors
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>> "Vector" has a more general sense than ML and
> > > > >>>> it's
> > > > >>>>>>> quite
> > > > >>>>>>>>>>> trivial
> > > > >>>>>>>>>>>> to
> > > > >>>>>>>>>>>>>>>>> produce vectors with non-finite values using
> > > > >>> e.g.
> > > > >>>>>> NumPy
> > > > >>>>>>>> or
> > > > >>>>>>>>>>> Pandas
> > > > >>>>>>>>>>>>> (or
> > > > >>>>>>>>>>>>>>>>> any other piece of dataframe or scientific
> > > > >>>>> computing
> > > > >>>>>>>>>> software).
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>> It's not like the general concept of a vector
> > > > >>> is
> > > > >>>> a
> > > > >>>>>> IETF
> > > > >>>>>>>>>>> standard
> > > > >>>>>>>>>>>> or
> > > > >>>>>>>>>>>>>>>>> anything, so analogies with JSON or UTF8
> > > > >> (where
> > > > >>>> the
> > > > >>>>>>>> values
> > > > >>>>>>>>>> are
> > > > >>>>>>>>>>>>>> expected
> > > > >>>>>>>>>>>>>>>>> to be valid according to the respective
> > > > >> specs)
> > > > >>> do
> > > > >>>>> not
> > > > >>>>>>>>> apply.
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>> This discussion would IMHO have a better
> > > > >> chance
> > > > >>>> of
> > > > >>>>>>>>> converging
> > > > >>>>>>>>>>> if
> > > > >>>>>>>>>>>> we
> > > > >>>>>>>>>>>>>>>>> separated the two orthogonal concerns of:
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>> 1. representing fixed-length list/vector data
> > > > >>> in
> > > > >>>>>>> Parquet
> > > > >>>>>>>>>> (with
> > > > >>>>>>>>>>> a
> > > > >>>>>>>>>>>>>>>>> potential limitation to non-nested types)
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>> 2. representing the information that a given
> > > > >>>> column
> > > > >>>>>> (or
> > > > >>>>>>>>>> column
> > > > >>>>>>>>>>>>> chunk,
> > > > >>>>>>>>>>>>>>>>> etc.) does not contain non-finite values
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>> ... both of which are useful on their own
> > > > >>> right,
> > > > >>>>> but
> > > > >>>>>> do
> > > > >>>>>>>> not
> > > > >>>>>>>>>>> have
> > > > >>>>>>>>>>>> to
> > > > >>>>>>>>>>>>>> be
> > > > >>>>>>>>>>>>>>>>> conflated and bundled together.
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>> Regards
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>> Antoine.
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>
> > > > >>>>>>>>>
> > > > >>>>>>>>
> > > > >>>>>>>
> > > > >>>>>>
> > > > >>>>>
> > > > >>>>
> > > > >>>
> > > > >>
> > > >
> > > >
> > >
> >
>

Re: [DISCUSS] Numeric Vector type: physical representation (outcome of focused call)

[Will Edwards via dev <de...@parquet.apache.org>]

lol and so you see I asked an llm to proof read my reply and pasted hastily
:D   But I did write my reply.  You can even reverse-engineer my human
version from the edits :D

:embarrassed:
Will

On Mon, 19 Oct 2026 at 12:11, Will Edwards <we...@spotify.com> wrote:

> It reads well. Three small fixes:
>
> 1. **"back in May or so"** is ambiguous. The fixed-size-list-for-tensors
> framing is from Rok's opening post in May 2024. The May 2026 design doc was
> already mostly about performance and layout. "Originally" avoids anyone
> correcting the date.
> 2. **"Spark's sparse feature vectors routinely use NaN"** isn't quite
> right. The NaN-for-missing behaviour comes from `VectorAssembler` and
> `Imputer` and applies to Spark ML vectors generally, dense or sparse.
> "Spark's feature vectors, dense and sparse," fixes it.
> 3. There are typos: "Iirc" should be "IIRC", and "Spark, BigQuery and
> DuckDB and pretty much…" has a doubled "and".
>
> The last sentence of the invariants paragraph could be clearer. "Although
> that doesn't mean readers shouldn't check on read" has a double negative
> and is easy to misread. "Though careful readers will still check on read"
> is simpler.
>
> With those applied:
>
> ---
>
> Hi all,
>
> The scope of this discussion keeps moving :). IIRC it originally was about
> fixed-size lists for ML tensors, became a performance question, and is now
> about whether vectors are mathematically valid. Each step made sense, but I
> don't think we have consensus on where we've ended up?
>
> I think there is much useful work in what we left behind. Many ML vectors
> aren't embeddings for search. E.g. Spark's feature vectors, dense and
> sparse, routinely use NaN to mark missing values, and quantised formats
> pack codes with scales in ways Parquet can't describe today. ML-oriented
> formats like Lance are moving towards tensors, quantisation and indexes. A
> type that told engines how to interpret that kind of data would enable new
> things. An annotation on LIST<FLOAT> doesn't.
>
> My big concern with the current focus on embeddings needing a logical type
> is compatibility. Today LIST<FLOAT> is the common format for embeddings,
> and Spark, BigQuery, DuckDB and pretty much every other library I can think
> of offer distance functions on plain lists, so any engine can use any file.
> If engines start requiring a VECTOR annotation, existing files and anything
> rewritten by a tool that doesn't know the type stop working. If they don't
> require it, the annotation adds nothing. We've seen this before with
> strings written without the UTF8 annotation, which engines still have to
> cope with years later.
>
> On invariants, I agree some users may want the writer to check what they
> write as they write it, but that can be a writer option rather than a type.
> Readers that want evidence could get it from extra statistics or metadata,
> though careful readers will still check on read.
>
> If a finite fixed-length vector contract is useful to table formats,
> perhaps Iceberg is the place for it, with Parquet storing plain lists. I'd
> raise the same worry there, though: if tables start rejecting untagged
> columns, we've just moved the split up a layer.
>
> Best,
> Will
>
> On Sat, 17 Oct 2026 at 00:10, Ryan Blue <bl...@apache.org> wrote:
>
>> I agree with much of what Dan said, but I'll add some to this thread as
>> well.
>>
>> First, I want to address the arguments that were pointed out:
>> 1. Other vector databases have finiteness requirements
>> 2. "Vectors are the hot stuff right now" -- let's jump on the bandwagon
>> 3. A general solution will take too long, finiteness will make faster
>> progress
>> 4. To give a guarantee to readers that no INF/NaN/NULL will be in the
>> vector
>>
>> I don't think these represent our actual motivation or position, which
>> makes them function as a straw-man. This is probably not what was
>> intended, but I want to make sure that the misunderstanding
>> doesn't continue. For one thing, we aren't saying that we need finite
>> vectors because they are "hot stuff".
>>
>> We want to build tools that use them on top of Parquet. I think it should
>> also be clear that we aren't advocating for finite elements blindly just
>> because other engines do (cargo cult design!). There are real
>> interoperability issues if we choose to go a different direction so this
>> is
>> a "Chesterton's Fence" situation. It's important to understand why
>> those databases
>> made that choice. I've also pointed this out as confirmation that we're
>> going down a reasonable and established path.
>>
>> The third argument sounds like a simple misunderstanding. We're pursuing a
>> logical type right now because that's forward compatible and we want to
>> label the data properly and set expectations for the type. Physical type
>> changes break forward compatibility and need to be released in a new
>> format
>> version . I think this is the source of the confusion since it has been in
>> the same discussion, but the choice about finiteness is separate.
>>
>> That brings us to the last argument, whether there is value in writers
>> participating in guarantees. I think there's value in adding requirements
>> for writers. It's up to readers how they want to use those requirements,
>> but a writer requirement allows a policy of "garbage in garbage out"
>> rather
>> than being forced to handle bad data. Why would a reader need to run
>> validation on every element just in case a writer had a bug? And why
>> should
>> the potential for writer bugs mean that it has to be the reader's problem?
>> If a writer produces bad data, we can absolutely reject it and tell the
>> writer to fix the bug.
>>
>> I'm also puzzled by the assertion that writer guarantees have value, but
>> only if there are other, more specific guarantees in addition. This seems
>> to cede the point that a contract between readers and writers makes sense,
>> but only if it is a _specific_ contract. But the proposed additions don't
>> make sense in the same way.
>>
>> Take, for example, the suggestion to also disallow zero-length vectors. At
>> a glance this seems similar, but I think it assumes a specific context and
>> is better suited to be a constraint for some use cases rather than part of
>> what is considered valid for the type. That's because zero-length vectors
>> are valid vectors with a length. You can calculate Euclidean or Manhattan
>> distance and use them in many contexts. They don't work if you're
>> normalizing to a length of 1, but it's a choice to normalize and use a dot
>> product to calculate cosine distance. And at this point, we're talking
>> about the usefulness of restrictions for the type, not a Platonic ideal.
>>
>> I think the restriction that makes the most sense is that vectors have a
>> length and can be compared in distance calculations. 0-length vectors are
>> valid because they point somewhere and have a length. We shouldn't
>> restrict
>> uses to only cosine distance over normalized vectors. Those are use case
>> dependent choices and there are other valid ones. I also don't think that
>> the model used to produce a vector is part of the vector type, just like
>> the method used to produce a metric doesn't define its type: cosine
>> similarity produces values between -1 and 1, but we store it in a `float`,
>> not a `float(cosine)`.
>>
>> Why does it make sense to have a finiteness restriction as part of the
>> type? Because that's what a (mathematical) vector is. A simple search
>> gives
>> a pretty common definition: "A vector is an object that has both magnitude
>> and a direction". If a dimension/element is NaN, null, or infinite, the
>> vector doesn't point anywhere and it doesn't have a magnitude. Not only
>> can
>> it not be used, it doesn't meet the most basic definition of the type.
>>
>> Ryan
>>
>> On Thu, Oct 15, 2026 at 1:07 AM Jan Finis <jp...@gmail.com> wrote:
>>
>> > >
>> > >
>> > >
>> > Daniel,
>> >
>> > alright, so no physical optimizations intended.  We only want this for
>> > semantic reasons, not for any significant physical optimization. If
>> we're
>> > all on the same page here, that is good 👍.
>> >
>> > To phrase my remaining problem with this more concretely:
>> > *Let's say we add this type. What cool new stuff can we now do that we
>> > couldn't do before?*
>> > That is the litmus test that a type has to pass to be useful IMHO. And
>> here
>> > I can't come up with a tangible use case that wasn't possible before.
>> > Especially if we one day have fixed size lists in Parquet, the remaining
>> > niche for this type becomes so small, that the added complexity isn't
>> worth
>> > the benefit IMHO.
>> >
>> > As I guess most people on here are active maintainers of a Parquet
>> > implementation: Concretely, what would you be using this type for if we
>> had
>> > it? Would you allow math only on fields tagged with this type? What if
>> that
>> > math yields INF or NaN? Not allowed?
>> >
>> > Especially if we also discuss having both VECTOR and FINITE_VECTOR. How
>> > will you guys wire this up with the type systems of your engines? Do
>> your
>> > engines also have two types here? Do you type-decay a FINITE_VECTOR to a
>> > VECTOR once someone does a multiplication that could yield INF? By
>> itself,
>> > a FINITE_VECTOR sounds good. Holistically looked at from a type system
>> > perspective, it comes with a lot of complexity. Do the benefits really
>> > outweigh the complexity here? That's why I would rather have the bare
>> > minimum: A fixed size array (list) of floats or at least just a vector
>> > without finite. Then all these detailed questions go away.
>> >
>> > I'll take this one step further... Why is there any debate about this at
>> > > all? Why can't a type simply be added to the *specification* as an
>> > > administrative detail and be done?
>> >
>> >
>> > Andrew,
>> >
>> > Parquet is supposed to be a lingua franca of data. The specification is
>> the
>> > authoritative truth for this language and all implementations are
>> expected
>> > to implement as much as possible from this spec, so anything in the spec
>> > does causes implementation effort. So just adding anything to this spec
>> > without debate is not a good idea. It is even the other way around: We
>> need
>> > to have way higher rigor with the spec than we need to have with any
>> > implementations. Implementations can be changed, the spec needs to be
>> > stable.
>> >
>> > A tag-only type that doesn't come with any restrictions can be added
>> with
>> > little effort. But here we want to tie restrictions to it (no NaN/Inf),
>> so
>> > we would expect implementations to enforce these. So there is some
>> > implementation effort expected here.
>> >
>> > Also, even if we only wanted a tag type, I would still argue that you
>> > shouldn't add those without a debate. What if A wants XML, B wants
>> YAML, C
>> > wants quaternions, D wants address, E wants phone number. If we just
>> added
>> > any types that are "just tags", Parquet logical types would just become
>> a
>> > trash dump of any type that anyone ever thought to be useful. So
>> rigorous
>> > assessment of each new type candidate is necessary.
>> >
>> > Finally, many engines couple parquet with another type system (e.g.,
>> Arrow
>> > or SQL). So even with a tag only type, you at least still have the
>> > implementation burden of wiring this up with your engine's type system.
>> >
>> > So, just because logical types are "just" logical, that doesn't mean we
>> can
>> > add them without any rigor.
>> >
>> >
>> > A few naive questions:
>> > > 1. Would a physical definition with NANs & Inf+/- allow efficient
>> > > encodings like ALP?
>> > > 2. If the physical representation allows values that make encodings
>> like
>> > > ALP impossible, can a tighter Logical type allow different encodings?
>> > > 3. Can we RLE or binpack a vector?
>> >
>> >
>> > The simple general answer here is that all these things do not require a
>> > logical type. Writers can already do any such optimization if the actual
>> > data contains no NaNs & Infs. All these physical optimizations just
>> require
>> > no actual NaNs/Infs to be present in a page, they don't need a guarantee
>> > that the whole file won't contain any.
>> >
>> > Cheers,
>> > Jan
>> >
>> >
>> >
>> >
>> > Am Do., 15. Okt. 2026 um 04:28 Uhr schrieb <ku...@gmail.com>:
>> >
>> > > Hi Everyone,
>> > >
>> > > Glad to see a Vector type being added. I have a few use cases I want
>> to
>> > > pursue once we have an agreed upon definition.
>> > >
>> > > In regards to that definition discussion. I feel comfortable with the
>> > idea
>> > > of the semantic meaning Dan raises though I can’t shake a slight
>> > discomfort
>> > > (maybe FOMO) with limiting values in the physical definition.
>> > >
>> > > A few naive questions:
>> > > 1. Would a physical definition with NANs & Inf+/- allow efficient
>> > > encodings like ALP?
>> > > 2. If the physical representation allows values that make encodings
>> like
>> > > ALP impossible, can a tighter Logical type allow different encodings?
>> > > 3. Can we RLE or binpack a vector?
>> > >
>> > > Best Regards,
>> > > Kurtis C. Wright
>> > > > On Oct 14, 2026, at 12:45, Daniel Weeks <dw...@apache.org> wrote:
>> > > >
>> > > > Hey Jan,
>> > > >
>> > > > There's a lot that I disagree with respect to the framing of these
>> > > > arguments and how we should be thinking about introducing vector as
>> a
>> > > > logical type.  There's a repeated tendency to let the physical
>> > > > representation or performance drive discussion around types as
>> opposed
>> > to
>> > > > what it represents.
>> > > >
>> > > > In parquet, establishing semantic meaning is the purpose of logical
>> > > types.
>> > > > The logical type separates how to interpret a value from its
>> physical
>> > > > representation.  We can represent a timestamp with INT64 or even a
>> > string
>> > > > value, so why have a type for it? Even though a FIXED_SIZED_LIST and
>> > > VECTOR
>> > > > may have the exact same representation and performance, they have
>> > > different
>> > > > semantic meaning.  Databases and engines across the ecosystem
>> already
>> > > > recognize this differentiation and have separate representations.
>> The
>> > > > choice to separate these isn't because the discussion is too
>> tedious;
>> > > > they're fundamentally different things.
>> > > >
>> > > > The finite requirement is what makes a vector a reflection of what
>> it
>> > > > represents.  As we've discussed in the thread, non-finite values in
>> the
>> > > > representation violate the properties of a vector and the value
>> ceases
>> > to
>> > > > be a vector.  On the other hand, a 0 vector is still a vector (you
>> can
>> > > > argue the utility of it, but it is a valid vector).  Implementations
>> > can
>> > > > choose what they validate or how to handle execution errors; that's
>> an
>> > > > implementation concern, not a requirement.  Operations performed on
>> a
>> > > > vector depend on the context in which they are used, but the
>> > > representation
>> > > > doesn't need to solve scenarios where that application doesn't make
>> > > sense.
>> > > >
>> > > > The guarantees or context needed to use a vector do not need to be
>> > > inherent
>> > > > in the type for it to be a useful type.  We don't need the currency
>> > used
>> > > > for decimal or whether a floating point is a normalized value as
>> part
>> > of
>> > > > the type to make use of those types.  This conflates the use cases
>> with
>> > > the
>> > > > type system and unnecessarily attempts to draw the context into the
>> > type
>> > > > definition.  Similarly, you don't need to capture normalization,
>> model,
>> > > > etc. as part of the type for it to be useful.
>> > > >
>> > > > So the reason we want to add a vector type (or FINITE_VECTOR) is
>> > because
>> > > we
>> > > > want to store vectors with the semantic meaning and requirements
>> based
>> > on
>> > > > what they represent.
>> > > >
>> > > > -Dan
>> > > >
>> > > >
>> > > >> On Wed, Oct 14, 2026 at 1:22 AM Jan Finis <jp...@gmail.com>
>> wrote:
>> > > >>
>> > > >> A bit late to the party, but here we go:
>> > > >>
>> > > >> I have the strong sentiment that the FINITE_VECTOR type as it is
>> > > currently
>> > > >> being discussed will not be very useful and just increase
>> > implementation
>> > > >> complexity. The arguments for the type are all quite weak (details
>> see
>> > > >> below). The FINITE part doesn't seem to yield any value and is not
>> a
>> > > useful
>> > > >> restriction by itself. The vector (i.e. fixed length) part should
>> > > rather be
>> > > >> generalized instead of making a one-off thing. In general, the
>> > existing
>> > > >> types are already working well for uncompressed vectors. The real
>> > > challenge
>> > > >> that would bring real benefits is supporting quantization. This is
>> > where
>> > > >> you get orders of magnitude improvements to storage space and
>> > processing
>> > > >> speed.
>> > > >>
>> > > >> Let's first discuss the FINITE part. So far, the only arguments I
>> have
>> > > read
>> > > >> for this are:
>> > > >>
>> > > >> 1. many other vector databases have it
>> > > >> 2. vectors are the hot stuff right now, so Parquet better boast
>> having
>> > > >> "native" type support for vectors
>> > > >> 3. by focussing on this common special case, we can make rapid
>> > progress,
>> > > >> while a general solution would take too long to specify
>> > > >> 4. to give a guarantee to readers that no INF/NaN/NULL will be in
>> the
>> > > >> vector
>> > > >>
>> > > >> the first two arguments are quite weak by themselves. Vector
>> databases
>> > > are
>> > > >> very specialized engines. Such engines can of course be very
>> > > opinionated on
>> > > >> their data types. A general format like Parquet must instead weigh
>> > > whether
>> > > >> a special type is worth the added complexity over a more general,
>> > widely
>> > > >> applicable type. In this case, it is IMHO not at all. And just
>> having
>> > a
>> > > >> type called "vector" for advertisement reasons to make Parquet
>> appear
>> > > more
>> > > >> modern seems a non-goal in the first place.
>> > > >>
>> > > >> The third "time to market" argument is valid for systems that don't
>> > > need a
>> > > >> stable interface. It is a very dangerous attitude for formats like
>> > > Parquet
>> > > >> that must be stable long term and should have a consolidated,
>> concise,
>> > > >> non-redundant type system. For such long term storage formats, we
>> need
>> > > to
>> > > >> take the time to build the right thing, as any shortcuts we take
>> will
>> > > stay
>> > > >> as blemishes in the format forever. If I read the discussion
>> > correctly,
>> > > >> this is actually the main argument as of now: We have found that
>> it is
>> > > hard
>> > > >> to align on how a fixed size list type should look like, so we
>> want to
>> > > go
>> > > >> the easier route of a specialized type. This seems like a
>> > > >> dangerous shortcut fueled by feature recency bias. Ask yourself:
>> Would
>> > > we
>> > > >> want this, if Parquet already had a fixed size list type?
>> > > >>
>> > > >> So finally, the only technical argument is the fourth one: "It
>> signals
>> > > >> semantics and ensures that no bad data lands in the format". I see
>> two
>> > > big
>> > > >> problems with this:
>> > > >>
>> > > >> 1. Readers *still* need to test for NaN / INF because Parquet files
>> > may
>> > > be
>> > > >> corrupt. Nothing can stop a malicious or buggy writer from
>> physically
>> > > >> writing NaN or INF. *So you save no single line of code in a
>> reader.
>> > You
>> > > >> save no branches, as you still need to handle the INF/NaN case.* So
>> > this
>> > > >> doesn't help with performance at all. The only difference is, that
>> the
>> > > >> error message changes from "found an invalid INF value in the
>> data" to
>> > > >> "Your parquet file seems to be corrupt". It is actually quite the
>> > > opposite,
>> > > >> readers and writers get more complex having to handle this new
>> type.
>> > > >>
>> > > >> If you now argue that it's not about performance, it's about
>> defined
>> > > >> semantics and erroring out early in writers, then the second
>> problem
>> > > >> arises:
>> > > >>
>> > > >> 2. The proposed no-NaN/INF/NULL guarantee is just one of multiple
>> > > >> guarantees that you need for sensible vector processing. If "no
>> > > >> INF/NaN/NULL" was the only guarantee you needed, I would agree that
>> > > >> FINITE_VECTOR is useful, but it is by no means the only guarantee
>> you
>> > > need,
>> > > >> instead, depending on the distance metric, you also need the
>> following
>> > > >> guarantees. E.g.:
>> > > >>
>> > > >>   - No vector may be the 0 vector, otherwise, e.g., cosine
>> similarity
>> > is
>> > > >>   undefined. Cosine similarity is often used as default, so this is
>> > > >> actually
>> > > >>   quite important in practice.
>> > > >>   - If we want to use a dot product, then all vectors need to
>> > guarantee
>> > > a
>> > > >>   length of 1. This would actually be quite useful for performance
>> > > >> reasons.
>> > > >>   - The vectors must come from the same model as the vector used
>> for
>> > the
>> > > >>   search. Otherwise, the search yields a numeric result, but that
>> > result
>> > > >> has
>> > > >>   no sensible meaning.
>> > > >>
>> > > >> So, what use is a type that just guarantees part of the things you
>> > need
>> > > for
>> > > >> vector search but not others? You still need additional guarantees
>> in
>> > > >> addition to that type. What makes INF/NaN so important in
>> comparison
>> > to,
>> > > >> say, "no 0 vector present", that it warrants an own type, while the
>> > > other
>> > > >> guarantees do not? If we really want this type as a basis for
>> > > cross-engine
>> > > >> sharing of vectors, then it needs to give *all* guarantees.
>> > > >>
>> > > >> I'm not against a specialized vector type if it really carries
>> *all*
>> > > >> information to be useful by itself. If it carried all that
>> > information,
>> > > >> then a reading engine could indeed validate that only meaningful
>> > vector
>> > > >> searches are executed. But as long as it is just a half-baked
>> thing,
>> > it
>> > > is
>> > > >> not useful by itself and just a burden on each implementer and
>> > > maintainer.
>> > > >> If we need extra information anyway, why not just stick to a
>> general
>> > > type
>> > > >> like VECTOR (without FINITE).
>> > > >>
>> > > >> If you're a proponent of FINITE_VECTOR, ask yourself (or tell us)
>> what
>> > > >> advantage you will *really* get over just a (potentially fixed
>> size)
>> > > List
>> > > >> of Float? As discussed, it's not performance. It's also not a
>> semantic
>> > > >> marker that is sufficient by itself. It's just a small hint that
>> > allows
>> > > >> writers to error out once they see a NaN or INF in the input, while
>> > they
>> > > >> will still happily write a bogus 0 vector. This seems to be of zero
>> > > >> benefit.
>> > > >>
>> > > >> Anecdotally, we currently handle petabytes of vectors in
>> production in
>> > > >> Data360. We simply store them in a List of Float, with an optimized
>> > > R-Level
>> > > >> reader implementation that rapidly checks that all vectors are the
>> > same
>> > > >> length. This solution works just fine. The overhead of R-level
>> > decoding
>> > > is
>> > > >> barely measurable. The check is super fast and R-levels are tiny in
>> > > >> comparison to the payload vectors, so also the added I/O is
>> > negligible.
>> > > >> This is also why I'm so firm in my sentiment: We are running a
>> > > production
>> > > >> grade system processing trillions of vectors right now and see no
>> big
>> > > >> issues with the types that Parquet currently provides.
>> > > >>
>> > > >> As I don't just want to be a naysayer and point out problems with
>> the
>> > > >> current design, here are the directions that I would propose
>> indeed:
>> > > >>
>> > > >>   - a general fixed size list type. Don't do shortcuts just because
>> > the
>> > > >>   discussion is too tedious.
>> > > >>   - a vector type that indeed carries enough information to be
>> really
>> > > >>   useful for vector search by itself. (e.g., model name / version,
>> > > >> optional
>> > > >>   guarantees for no 0 vector / normalized vector length). With
>> such a
>> > > >> type,
>> > > >>   you could really do well defined vector searches even if the
>> writing
>> > > >> engine
>> > > >>   is completely different to the reading one.
>> > > >>   - Standardize quantizations. This will be a real game changer for
>> > > >>   performance.
>> > > >>
>> > > >> Cheers,
>> > > >> Jan
>> > > >>
>> > > >>
>> > > >>
>> > > >>
>> > > >>
>> > > >>
>> > > >>> Am Mi., 14. Okt. 2026 um 03:36 Uhr schrieb Gang Wu <
>> ustcwg@gmail.com
>> > >:
>> > > >>>
>> > > >>> I think separating them into FINITE_VECTOR and FIXED_LENGTH_LIST
>> is a
>> > > >> good
>> > > >>> idea, though I think renaming FINITE_VECTOR to VECTOR looks more
>> > > natural.
>> > > >>> Should we move forward with proposing FINITE_VECTOR/VECTOR first?
>> > > >>>
>> > > >>> On Tue, Oct 13, 2026 at 6:26 PM Alkis Evlogimenos via dev <
>> > > >>> dev@parquet.apache.org> wrote:
>> > > >>>
>> > > >>>> I suggest we go with VECTOR for the finite vector to keep the
>> name
>> > the
>> > > >>> same
>> > > >>>> as other engines/type systems. If we want a non-finite one in the
>> > > >> future
>> > > >>> we
>> > > >>>> can name that something else or we add fixed size list that will
>> > serve
>> > > >>> the
>> > > >>>> same purpose without a special name.
>> > > >>>>
>> > > >>>>
>> > > >>>> On Thu, Oct 8, 2026 at 10:20 PM Ryan Blue <bl...@apache.org>
>> wrote:
>> > > >>>>
>> > > >>>>> A logical type would:
>> > > >>>>> - Describe data for users and readers to understand what the
>> file
>> > > >>>> contains
>> > > >>>>> - Signal that repetition and definition levels will reflect a
>> fixed
>> > > >>>> number
>> > > >>>>> of elements, which implementations can use to optimize decoding
>> > > >>>>> - Guarantee that there are no null elements (because the type
>> > cannot
>> > > >> be
>> > > >>>>> applied to a list with optional elements)
>> > > >>>>> - [FINITE_VECTOR] Guarantee that no elements are NaN or Infinity
>> > > >> values
>> > > >>>> so
>> > > >>>>> that the vectors have a finite length and are suitable for
>> distance
>> > > >>>>> calculations
>> > > >>>>>
>> > > >>>>> The finite value restriction is a restriction on writers to
>> prevent
>> > > >> bad
>> > > >>>>> data from being written.
>> > > >>>>>
>> > > >>>>> Ignoring logical types that are not understood is a rule for
>> > > >> readers. A
>> > > >>>>> reader can handle vectors and pass it on to another layer. For
>> > > >>> instance,
>> > > >>>> a
>> > > >>>>> Parquet reader that doesn't support geometry passes values as
>> > > >> `binary`.
>> > > >>>>> That binary value can be passed to a geometry UDF, even if the
>> > > >>> underlying
>> > > >>>>> engine and Parquet reader doesn't have support.
>> > > >>>>>
>> > > >>>>> A tool that doesn't understand FINITE_VECTOR can similarly pass
>> a
>> > > >> list
>> > > >>> of
>> > > >>>>> floats along to a UDF. But it is still important to the user
>> that
>> > bad
>> > > >>>>> values are not present in the data. In your example, bad values
>> are
>> > > >>>> vectors
>> > > >>>>> that cause the centroid calculation to fail, like vectors with
>> > > >> infinite
>> > > >>>>> length.
>> > > >>>>>
>> > > >>>>> Ryan
>> > > >>>>>
>> > > >>>>> On Thu, Oct 8, 2026 at 12:54 PM Will Edwards via dev <
>> > > >>>>> dev@parquet.apache.org>
>> > > >>>>> wrote:
>> > > >>>>>
>> > > >>>>>> Could we clarify the use cases for a VECTOR annotation on
>> > > >>> LIST<FLOAT>?
>> > > >>>>>> Specifically, what does it make possible that's impossible
>> today,
>> > > >> and
>> > > >>>>> what
>> > > >>>>>> does it make easier that's currently hard?
>> > > >>>>>>
>> > > >>>>>> I'm struggling to find a strong answer. I'd like every engine
>> to
>> > > >>> offer
>> > > >>>>>> vector functions on lists of floats without needing a new
>> logical
>> > > >>> type.
>> > > >>>>>>
>> > > >>>>>> To be clear, I'm still enthusiastic about a vector type that
>> > > >>> describes
>> > > >>>>> how
>> > > >>>>>> to unpack and scale composite representations (quantised codes
>> > plus
>> > > >>>>> scales,
>> > > >>>>>> codebooks, etc.), in the direction ML is going. That adds
>> > > >> information
>> > > >>>>>> engines don't have. A label on LIST<FLOAT> doesn't: engines
>> > already
>> > > >>>>>> understand the physical type, and many already offer vector
>> > > >> functions
>> > > >>>> on
>> > > >>>>>> it. I don't see a performance benefit, and NaN/null/Inf checks
>> are
>> > > >>>> cheap
>> > > >>>>>> enough that engines should do them anyway. The label also
>> doesn't
>> > > >> say
>> > > >>>>> what
>> > > >>>>>> you can do with the column. Knowing that a column holds
>> embeddings
>> > > >>>>> doesn't
>> > > >>>>>> say how they can be used, just as zipcode and age are both
>> > integers
>> > > >>> but
>> > > >>>>>> multiplying them is meaningless.
>> > > >>>>>>
>> > > >>>>>> My concern is the opposite effect. We're establishing that
>> engines
>> > > >>>> should
>> > > >>>>>> ignore logical types they don't understand. So a tool that
>> doesn't
>> > > >>> know
>> > > >>>>>> VECTOR will read the column as LIST<FLOAT>, let the user
>> compute
>> > > >>>>> centroids
>> > > >>>>>> (say), and write plain LIST<FLOAT> back out. If other engines
>> > > >>> restrict
>> > > >>>>>> embedding functions to VECTOR-annotated columns, that file
>> stops
>> > > >>>> working
>> > > >>>>>> with them, through no fault of the user. An annotation that
>> > doesn't
>> > > >>>>> change
>> > > >>>>>> what the data is shouldn't be able to make it unusable.
>> > > >>>>>>
>> > > >>>>>> Am I missing a use case?
>> > > >>>>>>
>> > > >>>>>> Best, Will
>> > > >>>>>>
>> > > >>>>>> On Thu, 8 Oct 2026 at 19:53, Steven Wu <st...@gmail.com>
>> > > >> wrote:
>> > > >>>>>>
>> > > >>>>>>> I also think a FINITE_VECTOR type brings a lot of values to
>> many
>> > > >>> use
>> > > >>>>>> cases.
>> > > >>>>>>> It is would be great to make progress on the FINITE_VECTOR,
>> and
>> > > >> not
>> > > >>>> to
>> > > >>>>>> fit
>> > > >>>>>>> every use cases into the FIXED_LENGTH_LIST.
>> > > >>>>>>>
>> > > >>>>>>> On Mon, Oct 5, 2026 at 1:26 PM Ryan Blue <bl...@apache.org>
>> > > >> wrote:
>> > > >>>>>>>
>> > > >>>>>>>> Going through this thread again, it looks like there are a
>> few
>> > > >>>> things
>> > > >>>>>>> that
>> > > >>>>>>>> people generally agree on:
>> > > >>>>>>>>
>> > > >>>>>>>> - In the short term, we want to introduce a logical type or
>> > > >>> types;
>> > > >>>> a
>> > > >>>>>>> better
>> > > >>>>>>>> physical representation can be done later. We can focus on
>> > > >>> logical
>> > > >>>>>> types.
>> > > >>>>>>>> - There is precedent in logical types for restricting values,
>> > > >>> like
>> > > >>>>> UTF8
>> > > >>>>>>>> strings to contain only UTF-8 codepoints. I think we also
>> agree
>> > > >>>> that
>> > > >>>>> we
>> > > >>>>>>>> could restrict vector elements to non-nested or primitive
>> > > >> types.
>> > > >>>>>>>> - There are many examples of databases that restrict their
>> > > >> vector
>> > > >>>>> types
>> > > >>>>>>> to
>> > > >>>>>>>> non-null, finite elements.
>> > > >>>>>>>> - Some people want to have write-time guarantees that vectors
>> > > >>> have
>> > > >>>>> only
>> > > >>>>>>>> non-null, finite elements
>> > > >>>>>>>> - Some people want to extend this work to include tensors,
>> > > >>> shorter
>> > > >>>>>>> floating
>> > > >>>>>>>> point representations, variable lengths, NaN or Infinity
>> > > >>> elements,
>> > > >>>>> etc.
>> > > >>>>>>>>
>> > > >>>>>>>> It sounds like we are nearly there, but have two halves of
>> the
>> > > >>>>>> community
>> > > >>>>>>>> that care about different things. That just means we should
>> > > >> have
>> > > >>>>>> multiple
>> > > >>>>>>>> types. It's entirely reasonable to want a type that allows
>> NaN
>> > > >>>>> values,
>> > > >>>>>>> and
>> > > >>>>>>>> it's also reasonable for those of us building databases,
>> table
>> > > >>>>>> standards,
>> > > >>>>>>>> and vector indexes to care about guarantees that elements are
>> > > >>>> finite.
>> > > >>>>>>>> There's more than enough examples to show that this is a
>> > > >>> legitimate
>> > > >>>>> use
>> > > >>>>>>>> case.
>> > > >>>>>>>>
>> > > >>>>>>>> Since I don't see the concerns of the different use cases
>> > > >>> changing,
>> > > >>>>> it
>> > > >>>>>>>> looks like we need to find a compromise to move forward. I
>> > > >>> propose
>> > > >>>>> that
>> > > >>>>>>> we
>> > > >>>>>>>> introduce a FINITE_VECTOR type and separately a
>> > > >> FIXED_LENGTH_LIST
>> > > >>>> or
>> > > >>>>>>>> another VECTOR type that allows infinite and NaN values.
>> > > >>>>>>>>
>> > > >>>>>>>> Can we agree on a compromise? Is there a strong reason to try
>> > > >> to
>> > > >>>> fit
>> > > >>>>>>> these
>> > > >>>>>>>> use cases into a single type? What if that means not making
>> > > >>>> progress?
>> > > >>>>>>>>
>> > > >>>>>>>> Ryan
>> > > >>>>>>>>
>> > > >>>>>>>> On Mon, Oct 5, 2026 at 12:15 PM Philipp Fischbeck via dev <
>> > > >>>>>>>> dev@parquet.apache.org> wrote:
>> > > >>>>>>>>
>> > > >>>>>>>>> Thinking about this more, I am also fine with pulling out
>> the
>> > > >>>>>> “NaN/Inf”
>> > > >>>>>>>>> requirement. Either into a type boolean (my preference), or
>> > > >>>> putting
>> > > >>>>>> no
>> > > >>>>>>>>> restrictions on the Parquet level, allowing all IEEE 754
>> > > >> float
>> > > >>>>>> numbers.
>> > > >>>>>>>> If
>> > > >>>>>>>>> some top-level application wants to put additional
>> > > >> constraints
>> > > >>> on
>> > > >>>>> the
>> > > >>>>>>>> data
>> > > >>>>>>>>> type, like Iceberg not allowing NaN/infinity, it can still
>> do
>> > > >>> so
>> > > >>>>> one
>> > > >>>>>>>> layer
>> > > >>>>>>>>> above. It seems like this was the biggest point of
>> discussion
>> > > >>>>>>> remaining.
>> > > >>>>>>>>> All these options work for me, and we should move forward
>> > > >> with
>> > > >>>> this
>> > > >>>>>>>>> proposal.
>> > > >>>>>>>>>
>> > > >>>>>>>>> Concerning additional element types like bf16, they can be
>> > > >>>>> introduced
>> > > >>>>>>> as
>> > > >>>>>>>>> Parquet types (at least in the current LIST-based spec
>> > > >>> proposal).
>> > > >>>>>>>>>
>> > > >>>>>>>>> Concerning Will’s suggestion around representing VECTOR as a
>> > > >>>>> variable
>> > > >>>>>>>>> length list with a MIN/MAX_LENGTH metadata restriction: I
>> > > >> think
>> > > >>>>> that
>> > > >>>>>> a
>> > > >>>>>>>>> dedicated logical VECTOR type still provides enough benefits
>> > > >> by
>> > > >>>>>>>> describing
>> > > >>>>>>>>> the semantics of the data, and allowing to capture future
>> > > >>> changes
>> > > >>>>>>> around
>> > > >>>>>>>>> statistics. In that case, the length is better captured as
>> an
>> > > >>>> exact
>> > > >>>>>>>> length
>> > > >>>>>>>>> for this type specifically. I’m also not sure about other
>> use
>> > > >>>> cases
>> > > >>>>>> for
>> > > >>>>>>>>> MIN_LENGTH, so introducing a new type constraint system just
>> > > >>> for
>> > > >>>>> the
>> > > >>>>>>>>> MIN_LENGTH=MAX_LENGTH case for vectors doesn’t have enough
>> > > >>> value
>> > > >>>> in
>> > > >>>>>> my
>> > > >>>>>>>>> view.
>> > > >>>>>>>>>
>> > > >>>>>>>>> Best,
>> > > >>>>>>>>> Philipp
>> > > >>>>>>>>>
>> > > >>>>>>>>>
>> > > >>>>>>>>> On Thu, Oct 1, 2026 at 9:00 AM Will Edwards via dev <
>> > > >>>>>>>>> dev@parquet.apache.org>
>> > > >>>>>>>>> wrote:
>> > > >>>>>>>>>
>> > > >>>>>>>>>> +1 Sebastian, a vector type suggests usage beyond
>> > > >> normalized
>> > > >>>>>>> embeddings
>> > > >>>>>>>>> and
>> > > >>>>>>>>>> engines really aren't going to be able to elide all the
>> > > >>> checks
>> > > >>>>>>> anyway.
>> > > >>>>>>>>>>
>> > > >>>>>>>>>> Antoine, you suggested we explore metadata solutions too :)
>> > > >>>>>>>>>>
>> > > >>>>>>>>>> Today, a database supporting CHAR(N) or VARCHAR(n) can't
>> > > >>>> express
>> > > >>>>>> this
>> > > >>>>>>>> in
>> > > >>>>>>>>>> Parquet.  Engines can write opaque blobs they understand in
>> > > >>> the
>> > > >>>>>>>> metadata
>> > > >>>>>>>>>> (e.g. Arrow does) so the extra type info round-trips, but
>> > > >>> this
>> > > >>>>>> isn't
>> > > >>>>>>>>>> standardized, portable, nor interoperable.
>> > > >>>>>>>>>>
>> > > >>>>>>>>>> So perhaps standardizing constraint metadata or new Thrift
>> > > >>>> fields
>> > > >>>>>> (my
>> > > >>>>>>>>> pref)
>> > > >>>>>>>>>> can generically support VECTOR(N) whilst also being more
>> > > >>>> broadly
>> > > >>>>>>> useful
>> > > >>>>>>>>> for
>> > > >>>>>>>>>> other use cases and data types?
>> > > >>>>>>>>>>
>> > > >>>>>>>>>> Imagine metadata or Thrift fields to tag the MIN_LENGTH and
>> > > >>>>>>> MAX_LENGTH
>> > > >>>>>>>>> for
>> > > >>>>>>>>>> Parquet's variable-length physcial types.  These would
>> > > >> count
>> > > >>>>>> logical
>> > > >>>>>>>>> units
>> > > >>>>>>>>>> (e.g. UTF8 code points for STRING) rather than bytes.
>> > > >>>>>>>>>>
>> > > >>>>>>>>>> Then one engine's VECTOR(N) can be written to Parquet as
>> > > >>>> LIST(N)
>> > > >>>>>> with
>> > > >>>>>>>>>> MIN_LENGTH = MAX_LENGTH = N and then readers - even
>> > > >> different
>> > > >>>>>>> engines -
>> > > >>>>>>>>> can
>> > > >>>>>>>>>> pick up these constraints if they support doing so.  We
>> > > >> don't
>> > > >>>>>>> actually
>> > > >>>>>>>>> need
>> > > >>>>>>>>>> a logical type to express a VECTOR then.
>> > > >>>>>>>>>>
>> > > >>>>>>>>>> This also allows standard SQL's CHAR(N) and VARCHAR(N) to
>> > > >> be
>> > > >>>>>>> expressed
>> > > >>>>>>>> on
>> > > >>>>>>>>>> UTF8 columns etc.
>> > > >>>>>>>>>>
>> > > >>>>>>>>>> This also seems generally useful in the future as we
>> > > >> approach
>> > > >>>>>>>> expressing
>> > > >>>>>>>>>> other tensor data shapes and types.
>> > > >>>>>>>>>>
>> > > >>>>>>>>>> This won't change the mechanics: the storage type doesn't
>> > > >>>> enforce
>> > > >>>>>>> these
>> > > >>>>>>>>>> constraints and engines must check them. It's previously
>> > > >>>>>> established
>> > > >>>>>>>> that
>> > > >>>>>>>>>> performance is just a software problem and there is no
>> > > >>>>> fundamental
>> > > >>>>>>>>>> performance advantage to the fixed size types in
>> > > >> Dremel-like
>> > > >>>>>> formats
>> > > >>>>>>>> like
>> > > >>>>>>>>>> Parquet.
>> > > >>>>>>>>>>
>> > > >>>>>>>>>> Best,
>> > > >>>>>>>>>> Will
>> > > >>>>>>>>>>
>> > > >>>>>>>>>> On Wed, 30 Sept 2026 at 16:08, Sebastian Baunsgaard <
>> > > >>>>>>>>> baunsgaard@apache.org
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>> wrote:
>> > > >>>>>>>>>>
>> > > >>>>>>>>>>> After listening in on the last meeting Wednesday, I have
>> > > >>> been
>> > > >>>>>>>> thinking
>> > > >>>>>>>>>>> about this quite a bit, and thought I would share my 2
>> > > >>> cents.
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>> I suggest not requiring VECTOR elements to be finite, as
>> > > >>>>> proposed
>> > > >>>>>>> in
>> > > >>>>>>>>> the
>> > > >>>>>>>>>> PR
>> > > >>>>>>>>>>> <https://github.com/apache/parquet-format/pull/624>.
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>> While Divjot and others note the specialization for
>> > > >> UTF-8,
>> > > >>>>>> Parquet
>> > > >>>>>>>>>> already
>> > > >>>>>>>>>>> defines DOUBLE and FLOAT as IEEE encodings
>> > > >>>>>>>>>>> <
>> > > >>>>>>>>
>> > > >>>>
>> > https://github.com/apache/parquet-format/blob/master/README.md#types
>> > > >>>>>>>>>> .
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>> Making NaN or +/-Inf illegal inside VECTOR would be a new
>> > > >>>>>>> restriction
>> > > >>>>>>>>> on
>> > > >>>>>>>>>>> those same underlying physical types. Scalar FLOAT/DOUBLE
>> > > >>>>> columns
>> > > >>>>>>>> have
>> > > >>>>>>>>>>> never had such a rule.
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>> It would also not match how the systems that actually
>> > > >>> produce
>> > > >>>>>> these
>> > > >>>>>>>>>> vectors
>> > > >>>>>>>>>>> treat floats. Most of them use IEEE floats, e.g. Spark,
>> > > >>>> DuckDB,
>> > > >>>>>>>> NumPy,
>> > > >>>>>>>>>>> PyTorch, TensorFlow, or lower level CUDA, oneMKL, and
>> > > >>>> OpenBLAS.
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>> pgvector's "all elements must be finite" rule is an
>> > > >> insert
>> > > >>>>>> policy.
>> > > >>>>>>>>> While
>> > > >>>>>>>>>> I
>> > > >>>>>>>>>>> think Parquet as a general-purpose format should allow
>> > > >> them
>> > > >>>> out
>> > > >>>>>> of
>> > > >>>>>>>> the
>> > > >>>>>>>>>> box.
>> > > >>>>>>>>>>> It is a user system decision to disallow writing these
>> > > >>> values
>> > > >>>>> or
>> > > >>>>>> to
>> > > >>>>>>>>> throw
>> > > >>>>>>>>>>> errors when encountering them.
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>> A type-level ban also does not keep Inf/NaN out of vector
>> > > >>>> math.
>> > > >>>>>>>> Finite
>> > > >>>>>>>>>>> coordinates still overflow. For instance: magnitude =
>> > > >>>>>> ||[FLOAT_MAX,
>> > > >>>>>>>>>>> FLOAT_MAX]|| where every coordinate is finite overflows
>> > > >> to
>> > > >>>>> +Inf.
>> > > >>>>>>>>> PyTorch
>> > > >>>>>>>>>>> documents the same overflow: tensor([1e20, 1e20]).norm()
>> > > >> is
>> > > >>>> inf
>> > > >>>>>> in
>> > > >>>>>>>>> fp32.
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>> A generated MAGNITUDE(vec) column would then be Inf while
>> > > >>> the
>> > > >>>>>>> VECTOR
>> > > >>>>>>>>>> itself
>> > > >>>>>>>>>>> was required to be finite. The ban only forbids storing
>> > > >>>>> specials
>> > > >>>>>> in
>> > > >>>>>>>> the
>> > > >>>>>>>>>>> coordinates, and does not prevent even simple downstream
>> > > >>>>>>> computations
>> > > >>>>>>>>>> from
>> > > >>>>>>>>>>> producing them.
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>> I suggest keeping VECTOR as the type and add an optional
>> > > >>> bool
>> > > >>>>>>>>>>> VectorType.finite (default false). Engines that want
>> > > >>>>>> pgvector-like
>> > > >>>>>>>>>>> rejection can require the flag = true. Engines that use
>> > > >>>> NumPy /
>> > > >>>>>>>>> PyTorch /
>> > > >>>>>>>>>>> Spark arrays can keep representable IEEE values. For the
>> > > >>> same
>> > > >>>>>>>> reason, I
>> > > >>>>>>>>>> do
>> > > >>>>>>>>>>> not favor adding yet another type called FINITE_VECTOR.
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>> Best,
>> > > >>>>>>>>>>> Sebastian
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>> On Tue, Sep 29, 2026 at 2:03 PM Divjot Arora via dev <
>> > > >>>>>>>>>>> dev@parquet.apache.org>
>> > > >>>>>>>>>>> wrote:
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>>> I agree with moving forward with the proposal and
>> > > >>> renaming
>> > > >>>>>> VECTOR
>> > > >>>>>>>> to
>> > > >>>>>>>>>>>> FINITE_VECTOR or something
>> > > >>>>>>>>>>>> similar if others feel that better communicates its
>> > > >>>> intent. I
>> > > >>>>>> see
>> > > >>>>>>>>> value
>> > > >>>>>>>>>>> in
>> > > >>>>>>>>>>>> having a vector type that guarantees
>> > > >>>>>>>>>>>> its values are mathematically-correct vectors for a few
>> > > >>>>>> reasons:
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>> 1. As others have mentioned, this is similar to the
>> > > >>> STRING
>> > > >>>>>>> logical
>> > > >>>>>>>>> type
>> > > >>>>>>>>>>>> mandating UTF-8.
>> > > >>>>>>>>>>>> 2. It is not possible to rely on statistics for
>> > > >> detecting
>> > > >>>> NaN
>> > > >>>>>> or
>> > > >>>>>>>>> +-inf
>> > > >>>>>>>>>>>> values as all statistics are optional.
>> > > >>>>>>>>>>>> To my knowledge, there is no precedent for requiring
>> > > >> that
>> > > >>>>>> writers
>> > > >>>>>>>>> emit
>> > > >>>>>>>>>>>> stats for specific logical types.
>> > > >>>>>>>>>>>> 3. Without statistics, NaN/inf values could flow
>> > > >> through
>> > > >>>> the
>> > > >>>>>>> engine
>> > > >>>>>>>>> and
>> > > >>>>>>>>>>>> block them for specific functionality.
>> > > >>>>>>>>>>>> This seems to be Lance's behavior. However, I don't
>> > > >> think
>> > > >>>>> this
>> > > >>>>>> is
>> > > >>>>>>>> as
>> > > >>>>>>>>>> easy
>> > > >>>>>>>>>>>> as it sounds and Lance has run
>> > > >>>>>>>>>>>> into several bugs due to this ([1] [2] for examples).
>> > > >>> Their
>> > > >>>>>>> Python
>> > > >>>>>>>>> [3]
>> > > >>>>>>>>>>> and
>> > > >>>>>>>>>>>> Rust [4] APIs default to erroring on
>> > > >>>>>>>>>>>> vectors that contain NaN values.
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>> I agree that Parquet is a general-purpose format, but I
>> > > >>>> don't
>> > > >>>>>>> think
>> > > >>>>>>>>>> this
>> > > >>>>>>>>>>>> implies the lack of domain-specific types.
>> > > >>>>>>>>>>>> Instead, this broadens the use cases where Parquet
>> > > >> would
>> > > >>> be
>> > > >>>>>>>>> considered
>> > > >>>>>>>>>> a
>> > > >>>>>>>>>>>> good fit. I also fully agree with
>> > > >>>>>>>>>>>> continuing work on a general fixed-size list type if we
>> > > >>>>> believe
>> > > >>>>>>>> there
>> > > >>>>>>>>>> are
>> > > >>>>>>>>>>>> non-vector use cases that would benefit
>> > > >>>>>>>>>>>> from such guarantees at the storage layer.
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>> [1] https://github.com/lance-format/lance/pull/3648
>> > > >>>>>>>>>>>> [2] https://github.com/lance-format/lance/issues/8607
>> > > >>> (see
>> > > >>>>>>> point 2
>> > > >>>>>>>>> in
>> > > >>>>>>>>>>>> "root
>> > > >>>>>>>>>>>> cause")
>> > > >>>>>>>>>>>> [3]
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>
>> > > >>>>>>>>>
>> > > >>>>>>>>
>> > > >>>>>>>
>> > > >>>>>>
>> > > >>>>>
>> > > >>>>
>> > > >>>
>> > > >>
>> > >
>> >
>> https://github.com/lancedb/lancedb/blob/0327618a62483feb6dca2336549845e5887cd48f/python/python/lancedb/table.py#L4807-L4911
>> > > >>>>>>>>>>>> [4]
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>
>> > > >>>>>>>>>
>> > > >>>>>>>>
>> > > >>>>>>>
>> > > >>>>>>
>> > > >>>>>
>> > > >>>>
>> > > >>>
>> > > >>
>> > >
>> >
>> https://github.com/lancedb/lancedb/blob/0327618a62483feb6dca2336549845e5887cd48f/rust/lancedb/src/table/add_data.rs#L105-L110
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>> On Mon, Sep 28, 2026 at 6:19 PM Philipp Fischbeck via
>> > > >>> dev <
>> > > >>>>>>>>>>>> dev@parquet.apache.org> wrote:
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>>> I also favor moving forward with the proposal as is.
>> > > >> I
>> > > >>>> see
>> > > >>>>>>> strong
>> > > >>>>>>>>>> value
>> > > >>>>>>>>>>>> and
>> > > >>>>>>>>>>>>> use cases for the numeric vector type, and I agree
>> > > >> with
>> > > >>>> the
>> > > >>>>>>>>>>> restrictions
>> > > >>>>>>>>>>>>> around NaN/Inf values and element types in general.
>> > > >>>>> Precedent
>> > > >>>>>>>>> exists
>> > > >>>>>>>>>> in
>> > > >>>>>>>>>>>>> other types that restrict which physical type values
>> > > >>> are
>> > > >>>>>>>> logically
>> > > >>>>>>>>>>> valid
>> > > >>>>>>>>>>>>> (e.g., the geospatial types needing to be
>> > > >> WKB-encoded).
>> > > >>>> A
>> > > >>>>>>>> general
>> > > >>>>>>>>>>>>> fixed-size list also has value, but I think it makes
>> > > >>>> sense
>> > > >>>>> to
>> > > >>>>>>>>>> separate
>> > > >>>>>>>>>>>> the
>> > > >>>>>>>>>>>>> two types. The semantics are relevant, and I can see
>> > > >>> how
>> > > >>>>> the
>> > > >>>>>>> two
>> > > >>>>>>>>>> types
>> > > >>>>>>>>>>>>> might diverge further in the future regarding
>> > > >>> statistics
>> > > >>>> or
>> > > >>>>>>>>>> encodings.
>> > > >>>>>>>>>>>>>
>> > > >>>>>>>>>>>>> I would be okay with renaming the vector type, but it
>> > > >>>> seems
>> > > >>>>>>> like
>> > > >>>>>>>>> the
>> > > >>>>>>>>>>>> VECTOR
>> > > >>>>>>>>>>>>> name is already used across other engines mentioned
>> > > >> in
>> > > >>>> this
>> > > >>>>>>>> thread.
>> > > >>>>>>>>>>>>>
>> > > >>>>>>>>>>>>> Best,
>> > > >>>>>>>>>>>>> Philipp
>> > > >>>>>>>>>>>>>
>> > > >>>>>>>>>>>>> On Sat, Sep 26, 2026 at 4:37 PM Gang Wu <
>> > > >>>> ustcwg@gmail.com>
>> > > >>>>>>>> wrote:
>> > > >>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>> To separate orthogonal concerns, I feel we should
>> > > >>> treat
>> > > >>>>> the
>> > > >>>>>>>>> general
>> > > >>>>>>>>>>>>>> FixedSizeList as the underlying physical layout,
>> > > >> and
>> > > >>>>>> separate
>> > > >>>>>>>> it
>> > > >>>>>>>>>> from
>> > > >>>>>>>>>>>> the
>> > > >>>>>>>>>>>>>> discussion of a specialized VECTOR (or broader ML
>> > > >>>> tensor)
>> > > >>>>>>>> logical
>> > > >>>>>>>>>>> type.
>> > > >>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>> Currently, the immediate need is to define a
>> > > >>>> specialized
>> > > >>>>>>> VECTOR
>> > > >>>>>>>>>> type
>> > > >>>>>>>>>>> so
>> > > >>>>>>>>>>>>>> that different engines and formats have a common
>> > > >>>> semantic
>> > > >>>>>>>>> contract
>> > > >>>>>>>>>>> for
>> > > >>>>>>>>>>>>>> interoperability.
>> > > >>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>> If the main point of contention is just the
>> > > >>>> NaN/Inf/null
>> > > >>>>>>>>>> limitations,
>> > > >>>>>>>>>>>>> maybe
>> > > >>>>>>>>>>>>>> we can resolve this by adding a qualification to
>> > > >> the
>> > > >>>> name
>> > > >>>>>>> (like
>> > > >>>>>>>>>>> Andrew
>> > > >>>>>>>>>>>>>> suggested, e.g., VECTOR_EMBEDDING or FINITE_VECTOR)
>> > > >>> to
>> > > >>>>>> avoid
>> > > >>>>>>>>>>> confusion
>> > > >>>>>>>>>>>>> with
>> > > >>>>>>>>>>>>>> broader ML use cases. Alternatively, we could
>> > > >>>> introduce a
>> > > >>>>>>>> boolean
>> > > >>>>>>>>>>> flag
>> > > >>>>>>>>>>>>>> (e.g., optional bool VectorType.finite) as proposed
>> > > >>>>>> earlier.
>> > > >>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>> On Fri, Sep 25, 2026 at 3:52 AM Andrew Lamb <
>> > > >>>>>>>>>> andrewlamb11@gmail.com>
>> > > >>>>>>>>>>>>>> wrote:
>> > > >>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>> I am a little confused about the points of
>> > > >>>> disagreement
>> > > >>>>>>> here.
>> > > >>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>> Is it that the name `VECTOR` is too general for
>> > > >> the
>> > > >>>>>>> specific
>> > > >>>>>>>>> use
>> > > >>>>>>>>>>> case
>> > > >>>>>>>>>>>>> of
>> > > >>>>>>>>>>>>>>> fixed length arrays with non-finite values used
>> > > >> for
>> > > >>>>>> vector
>> > > >>>>>>>>>>>> embeddings?
>> > > >>>>>>>>>>>>>> If
>> > > >>>>>>>>>>>>>>> so, maybe we could add a qualification to the
>> > > >> name
>> > > >>>>>>>>>> (FINITE_VECTOR?
>> > > >>>>>>>>>>>>>>> FEATURE_VECTOR? VECTOR_EMBEDDING?)
>> > > >>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>> It seems like there is precedent for defining
>> > > >>> logical
>> > > >>>>>> types
>> > > >>>>>>>>> that
>> > > >>>>>>>>>>>>> specify
>> > > >>>>>>>>>>>>>>> certain physical types and restrict their
>> > > >> contents
>> > > >>> to
>> > > >>>>>> help
>> > > >>>>>>>>>> readers
>> > > >>>>>>>>>>>>>>> interpret meaning, for example string[1] or JSON;
>> > > >>>>>> Defining
>> > > >>>>>>>> one
>> > > >>>>>>>>>> for
>> > > >>>>>>>>>>>>>> vector
>> > > >>>>>>>>>>>>>>> embeddings seems reasonable enough to me.
>> > > >>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>> Andrew
>> > > >>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>> [1]:
>> > > >>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>
>> > > >>>>>>>>>
>> > > >>>>>>>>
>> > > >>>>>>>
>> > > >>>>>>
>> > > >>>>>
>> > > >>>>
>> > > >>>
>> > > >>
>> > >
>> >
>> https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#string
>> > > >>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>> On Thu, Sep 24, 2026 at 3:34 PM Daniel Weeks <
>> > > >>>>>>>>> dweeks@apache.org>
>> > > >>>>>>>>>>>>> wrote:
>> > > >>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>> Well, Parquet is not a single-purpose format.
>> > > >>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>> Vector stores can choose whatever policy fits
>> > > >>>> their
>> > > >>>>>>>> usage,
>> > > >>>>>>>>>> but
>> > > >>>>>>>>>>>>> we're
>> > > >>>>>>>>>>>>>>>>> talking about a general-purpose file format
>> > > >>> that
>> > > >>>>> aims
>> > > >>>>>>> to
>> > > >>>>>>>> be
>> > > >>>>>>>>>>>> broadly
>> > > >>>>>>>>>>>>>>>>> applicable.
>> > > >>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>> I agree that the format is intended to be
>> > > >> general
>> > > >>>>>>> purpose,
>> > > >>>>>>>>> but
>> > > >>>>>>>>>>>> types
>> > > >>>>>>>>>>>>>>> within
>> > > >>>>>>>>>>>>>>>> the format define specific representation and
>> > > >>>>> semantics
>> > > >>>>>>> to
>> > > >>>>>>>>>>> achieve
>> > > >>>>>>>>>>>>> that
>> > > >>>>>>>>>>>>>>>> goal.
>> > > >>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>> "Vector" has a more general sense than ML and
>> > > >>>> it's
>> > > >>>>>>> quite
>> > > >>>>>>>>>>> trivial
>> > > >>>>>>>>>>>> to
>> > > >>>>>>>>>>>>>>>>> produce vectors with non-finite values using
>> > > >>> e.g.
>> > > >>>>>> NumPy
>> > > >>>>>>>> or
>> > > >>>>>>>>>>> Pandas
>> > > >>>>>>>>>>>>> (or
>> > > >>>>>>>>>>>>>>>>> any other piece of dataframe or scientific
>> > > >>>>> computing
>> > > >>>>>>>>>> software).
>> > > >>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>> The justification for restricting non-finite
>> > > >>>> values
>> > > >>>>> is
>> > > >>>>>>>>>> directly
>> > > >>>>>>>>>>>>>> related
>> > > >>>>>>>>>>>>>>> to
>> > > >>>>>>>>>>>>>>>> the semantic meaning of a vector.  Vectors have
>> > > >>>>>> specific
>> > > >>>>>>>>>>>>>> characteristics
>> > > >>>>>>>>>>>>>>>> (dimension, magnitude, direction, etc).
>> > > >>> Non-finite
>> > > >>>>>>> values
>> > > >>>>>>>>> are
>> > > >>>>>>>>>>>>> invalid
>> > > >>>>>>>>>>>>>>> and
>> > > >>>>>>>>>>>>>>>> break those characteristics.  I'd liken it to
>> > > >>>> storing
>> > > >>>>>>>>> non-utf8
>> > > >>>>>>>>>>>> values
>> > > >>>>>>>>>>>>>> in
>> > > >>>>>>>>>>>>>>> a
>> > > >>>>>>>>>>>>>>>> string and still calling it a string.
>> > > >>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>> That also differentiates a vector from a fixed
>> > > >>>> length
>> > > >>>>>>> list,
>> > > >>>>>>>>>> which
>> > > >>>>>>>>>>>> is
>> > > >>>>>>>>>>>>>> more
>> > > >>>>>>>>>>>>>>>> general and does not carry the same semantic
>> > > >>>> meaning.
>> > > >>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>> I also don't think we need to bundle all
>> > > >>> similarly
>> > > >>>>>> shaped
>> > > >>>>>>>> use
>> > > >>>>>>>>>>> cases
>> > > >>>>>>>>>>>>>> into
>> > > >>>>>>>>>>>>>>>> the same solution (e.g. flxed size list).
>> > > >> Vector
>> > > >>>> is
>> > > >>>>> a
>> > > >>>>>>> well
>> > > >>>>>>>>>>>>> established
>> > > >>>>>>>>>>>>>>>> data type at this point and aligns with the
>> > > >>> general
>> > > >>>>>>> purpose
>> > > >>>>>>>>>>> nature
>> > > >>>>>>>>>>>> of
>> > > >>>>>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>> format (many other formats and databases
>> > > >> natively
>> > > >>>>>> support
>> > > >>>>>>>>>> this).
>> > > >>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>> -Dan
>> > > >>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>> On Thu, Sep 24, 2026 at 11:55 AM Antoine
>> > > >> Pitrou <
>> > > >>>>>>>>>>>> antoine@python.org>
>> > > >>>>>>>>>>>>>>>> wrote:
>> > > >>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>> Le 24/09/2026 à 20:11, Russell Spitzer a
>> > > >> écrit
>> > > >>> :
>> > > >>>>>>>>>>>>>>>>>> I think something I haven't seen yet is the
>> > > >>>> clear
>> > > >>>>>>>>> use-case
>> > > >>>>>>>>>>> for
>> > > >>>>>>>>>>>>>>>> non-finite
>> > > >>>>>>>>>>>>>>>>>> values within a vector type. There has
>> > > >> been a
>> > > >>>> lot
>> > > >>>>>> of
>> > > >>>>>>>>>>>> theoretical
>> > > >>>>>>>>>>>>>>>>> discussion
>> > > >>>>>>>>>>>>>>>>>> here about use cases but what toolings
>> > > >>>> currently
>> > > >>>>>>>> produce
>> > > >>>>>>>>>>> these
>> > > >>>>>>>>>>>>>>> vectors
>> > > >>>>>>>>>>>>>>>>> and
>> > > >>>>>>>>>>>>>>>>>> how do they function in downstream engines?
>> > > >>> I'd
>> > > >>>>>> like
>> > > >>>>>>> to
>> > > >>>>>>>>>> limit
>> > > >>>>>>>>>>>>>>>> introducing
>> > > >>>>>>>>>>>>>>>>>> types that have a wide range of values
>> > > >> which
>> > > >>>> are
>> > > >>>>>>> simply
>> > > >>>>>>>>> are
>> > > >>>>>>>>>>>>> failure
>> > > >>>>>>>>>>>>>>>> modes
>> > > >>>>>>>>>>>>>>>>>> of different flavors
>> > > >>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>> "Vector" has a more general sense than ML and
>> > > >>>> it's
>> > > >>>>>>> quite
>> > > >>>>>>>>>>> trivial
>> > > >>>>>>>>>>>> to
>> > > >>>>>>>>>>>>>>>>> produce vectors with non-finite values using
>> > > >>> e.g.
>> > > >>>>>> NumPy
>> > > >>>>>>>> or
>> > > >>>>>>>>>>> Pandas
>> > > >>>>>>>>>>>>> (or
>> > > >>>>>>>>>>>>>>>>> any other piece of dataframe or scientific
>> > > >>>>> computing
>> > > >>>>>>>>>> software).
>> > > >>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>> It's not like the general concept of a vector
>> > > >>> is
>> > > >>>> a
>> > > >>>>>> IETF
>> > > >>>>>>>>>>> standard
>> > > >>>>>>>>>>>> or
>> > > >>>>>>>>>>>>>>>>> anything, so analogies with JSON or UTF8
>> > > >> (where
>> > > >>>> the
>> > > >>>>>>>> values
>> > > >>>>>>>>>> are
>> > > >>>>>>>>>>>>>> expected
>> > > >>>>>>>>>>>>>>>>> to be valid according to the respective
>> > > >> specs)
>> > > >>> do
>> > > >>>>> not
>> > > >>>>>>>>> apply.
>> > > >>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>> This discussion would IMHO have a better
>> > > >> chance
>> > > >>>> of
>> > > >>>>>>>>> converging
>> > > >>>>>>>>>>> if
>> > > >>>>>>>>>>>> we
>> > > >>>>>>>>>>>>>>>>> separated the two orthogonal concerns of:
>> > > >>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>> 1. representing fixed-length list/vector data
>> > > >>> in
>> > > >>>>>>> Parquet
>> > > >>>>>>>>>> (with
>> > > >>>>>>>>>>> a
>> > > >>>>>>>>>>>>>>>>> potential limitation to non-nested types)
>> > > >>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>> 2. representing the information that a given
>> > > >>>> column
>> > > >>>>>> (or
>> > > >>>>>>>>>> column
>> > > >>>>>>>>>>>>> chunk,
>> > > >>>>>>>>>>>>>>>>> etc.) does not contain non-finite values
>> > > >>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>> ... both of which are useful on their own
>> > > >>> right,
>> > > >>>>> but
>> > > >>>>>> do
>> > > >>>>>>>> not
>> > > >>>>>>>>>>> have
>> > > >>>>>>>>>>>> to
>> > > >>>>>>>>>>>>>> be
>> > > >>>>>>>>>>>>>>>>> conflated and bundled together.
>> > > >>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>> Regards
>> > > >>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>> Antoine.
>> > > >>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>
>> > > >>>>>>>>>
>> > > >>>>>>>>
>> > > >>>>>>>
>> > > >>>>>>
>> > > >>>>>
>> > > >>>>
>> > > >>>
>> > > >>
>> > >
>> > >
>> >
>>
>

Re: [DISCUSS] Numeric Vector type: physical representation (outcome of focused call)

[Will Edwards via dev <de...@parquet.apache.org>]

It reads well. Three small fixes:

1. **"back in May or so"** is ambiguous. The fixed-size-list-for-tensors
framing is from Rok's opening post in May 2024. The May 2026 design doc was
already mostly about performance and layout. "Originally" avoids anyone
correcting the date.
2. **"Spark's sparse feature vectors routinely use NaN"** isn't quite
right. The NaN-for-missing behaviour comes from `VectorAssembler` and
`Imputer` and applies to Spark ML vectors generally, dense or sparse.
"Spark's feature vectors, dense and sparse," fixes it.
3. There are typos: "Iirc" should be "IIRC", and "Spark, BigQuery and
DuckDB and pretty much…" has a doubled "and".

The last sentence of the invariants paragraph could be clearer. "Although
that doesn't mean readers shouldn't check on read" has a double negative
and is easy to misread. "Though careful readers will still check on read"
is simpler.

With those applied:

---

Hi all,

The scope of this discussion keeps moving :). IIRC it originally was about
fixed-size lists for ML tensors, became a performance question, and is now
about whether vectors are mathematically valid. Each step made sense, but I
don't think we have consensus on where we've ended up?

I think there is much useful work in what we left behind. Many ML vectors
aren't embeddings for search. E.g. Spark's feature vectors, dense and
sparse, routinely use NaN to mark missing values, and quantised formats
pack codes with scales in ways Parquet can't describe today. ML-oriented
formats like Lance are moving towards tensors, quantisation and indexes. A
type that told engines how to interpret that kind of data would enable new
things. An annotation on LIST<FLOAT> doesn't.

My big concern with the current focus on embeddings needing a logical type
is compatibility. Today LIST<FLOAT> is the common format for embeddings,
and Spark, BigQuery, DuckDB and pretty much every other library I can think
of offer distance functions on plain lists, so any engine can use any file.
If engines start requiring a VECTOR annotation, existing files and anything
rewritten by a tool that doesn't know the type stop working. If they don't
require it, the annotation adds nothing. We've seen this before with
strings written without the UTF8 annotation, which engines still have to
cope with years later.

On invariants, I agree some users may want the writer to check what they
write as they write it, but that can be a writer option rather than a type.
Readers that want evidence could get it from extra statistics or metadata,
though careful readers will still check on read.

If a finite fixed-length vector contract is useful to table formats,
perhaps Iceberg is the place for it, with Parquet storing plain lists. I'd
raise the same worry there, though: if tables start rejecting untagged
columns, we've just moved the split up a layer.

Best,
Will

On Sat, 17 Oct 2026 at 00:10, Ryan Blue <bl...@apache.org> wrote:

> I agree with much of what Dan said, but I'll add some to this thread as
> well.
>
> First, I want to address the arguments that were pointed out:
> 1. Other vector databases have finiteness requirements
> 2. "Vectors are the hot stuff right now" -- let's jump on the bandwagon
> 3. A general solution will take too long, finiteness will make faster
> progress
> 4. To give a guarantee to readers that no INF/NaN/NULL will be in the
> vector
>
> I don't think these represent our actual motivation or position, which
> makes them function as a straw-man. This is probably not what was
> intended, but I want to make sure that the misunderstanding
> doesn't continue. For one thing, we aren't saying that we need finite
> vectors because they are "hot stuff".
>
> We want to build tools that use them on top of Parquet. I think it should
> also be clear that we aren't advocating for finite elements blindly just
> because other engines do (cargo cult design!). There are real
> interoperability issues if we choose to go a different direction so this is
> a "Chesterton's Fence" situation. It's important to understand why
> those databases
> made that choice. I've also pointed this out as confirmation that we're
> going down a reasonable and established path.
>
> The third argument sounds like a simple misunderstanding. We're pursuing a
> logical type right now because that's forward compatible and we want to
> label the data properly and set expectations for the type. Physical type
> changes break forward compatibility and need to be released in a new format
> version . I think this is the source of the confusion since it has been in
> the same discussion, but the choice about finiteness is separate.
>
> That brings us to the last argument, whether there is value in writers
> participating in guarantees. I think there's value in adding requirements
> for writers. It's up to readers how they want to use those requirements,
> but a writer requirement allows a policy of "garbage in garbage out" rather
> than being forced to handle bad data. Why would a reader need to run
> validation on every element just in case a writer had a bug? And why should
> the potential for writer bugs mean that it has to be the reader's problem?
> If a writer produces bad data, we can absolutely reject it and tell the
> writer to fix the bug.
>
> I'm also puzzled by the assertion that writer guarantees have value, but
> only if there are other, more specific guarantees in addition. This seems
> to cede the point that a contract between readers and writers makes sense,
> but only if it is a _specific_ contract. But the proposed additions don't
> make sense in the same way.
>
> Take, for example, the suggestion to also disallow zero-length vectors. At
> a glance this seems similar, but I think it assumes a specific context and
> is better suited to be a constraint for some use cases rather than part of
> what is considered valid for the type. That's because zero-length vectors
> are valid vectors with a length. You can calculate Euclidean or Manhattan
> distance and use them in many contexts. They don't work if you're
> normalizing to a length of 1, but it's a choice to normalize and use a dot
> product to calculate cosine distance. And at this point, we're talking
> about the usefulness of restrictions for the type, not a Platonic ideal.
>
> I think the restriction that makes the most sense is that vectors have a
> length and can be compared in distance calculations. 0-length vectors are
> valid because they point somewhere and have a length. We shouldn't restrict
> uses to only cosine distance over normalized vectors. Those are use case
> dependent choices and there are other valid ones. I also don't think that
> the model used to produce a vector is part of the vector type, just like
> the method used to produce a metric doesn't define its type: cosine
> similarity produces values between -1 and 1, but we store it in a `float`,
> not a `float(cosine)`.
>
> Why does it make sense to have a finiteness restriction as part of the
> type? Because that's what a (mathematical) vector is. A simple search gives
> a pretty common definition: "A vector is an object that has both magnitude
> and a direction". If a dimension/element is NaN, null, or infinite, the
> vector doesn't point anywhere and it doesn't have a magnitude. Not only can
> it not be used, it doesn't meet the most basic definition of the type.
>
> Ryan
>
> On Thu, Oct 15, 2026 at 1:07 AM Jan Finis <jp...@gmail.com> wrote:
>
> > >
> > >
> > >
> > Daniel,
> >
> > alright, so no physical optimizations intended.  We only want this for
> > semantic reasons, not for any significant physical optimization. If we're
> > all on the same page here, that is good 👍.
> >
> > To phrase my remaining problem with this more concretely:
> > *Let's say we add this type. What cool new stuff can we now do that we
> > couldn't do before?*
> > That is the litmus test that a type has to pass to be useful IMHO. And
> here
> > I can't come up with a tangible use case that wasn't possible before.
> > Especially if we one day have fixed size lists in Parquet, the remaining
> > niche for this type becomes so small, that the added complexity isn't
> worth
> > the benefit IMHO.
> >
> > As I guess most people on here are active maintainers of a Parquet
> > implementation: Concretely, what would you be using this type for if we
> had
> > it? Would you allow math only on fields tagged with this type? What if
> that
> > math yields INF or NaN? Not allowed?
> >
> > Especially if we also discuss having both VECTOR and FINITE_VECTOR. How
> > will you guys wire this up with the type systems of your engines? Do your
> > engines also have two types here? Do you type-decay a FINITE_VECTOR to a
> > VECTOR once someone does a multiplication that could yield INF? By
> itself,
> > a FINITE_VECTOR sounds good. Holistically looked at from a type system
> > perspective, it comes with a lot of complexity. Do the benefits really
> > outweigh the complexity here? That's why I would rather have the bare
> > minimum: A fixed size array (list) of floats or at least just a vector
> > without finite. Then all these detailed questions go away.
> >
> > I'll take this one step further... Why is there any debate about this at
> > > all? Why can't a type simply be added to the *specification* as an
> > > administrative detail and be done?
> >
> >
> > Andrew,
> >
> > Parquet is supposed to be a lingua franca of data. The specification is
> the
> > authoritative truth for this language and all implementations are
> expected
> > to implement as much as possible from this spec, so anything in the spec
> > does causes implementation effort. So just adding anything to this spec
> > without debate is not a good idea. It is even the other way around: We
> need
> > to have way higher rigor with the spec than we need to have with any
> > implementations. Implementations can be changed, the spec needs to be
> > stable.
> >
> > A tag-only type that doesn't come with any restrictions can be added with
> > little effort. But here we want to tie restrictions to it (no NaN/Inf),
> so
> > we would expect implementations to enforce these. So there is some
> > implementation effort expected here.
> >
> > Also, even if we only wanted a tag type, I would still argue that you
> > shouldn't add those without a debate. What if A wants XML, B wants YAML,
> C
> > wants quaternions, D wants address, E wants phone number. If we just
> added
> > any types that are "just tags", Parquet logical types would just become a
> > trash dump of any type that anyone ever thought to be useful. So rigorous
> > assessment of each new type candidate is necessary.
> >
> > Finally, many engines couple parquet with another type system (e.g.,
> Arrow
> > or SQL). So even with a tag only type, you at least still have the
> > implementation burden of wiring this up with your engine's type system.
> >
> > So, just because logical types are "just" logical, that doesn't mean we
> can
> > add them without any rigor.
> >
> >
> > A few naive questions:
> > > 1. Would a physical definition with NANs & Inf+/- allow efficient
> > > encodings like ALP?
> > > 2. If the physical representation allows values that make encodings
> like
> > > ALP impossible, can a tighter Logical type allow different encodings?
> > > 3. Can we RLE or binpack a vector?
> >
> >
> > The simple general answer here is that all these things do not require a
> > logical type. Writers can already do any such optimization if the actual
> > data contains no NaNs & Infs. All these physical optimizations just
> require
> > no actual NaNs/Infs to be present in a page, they don't need a guarantee
> > that the whole file won't contain any.
> >
> > Cheers,
> > Jan
> >
> >
> >
> >
> > Am Do., 15. Okt. 2026 um 04:28 Uhr schrieb <ku...@gmail.com>:
> >
> > > Hi Everyone,
> > >
> > > Glad to see a Vector type being added. I have a few use cases I want to
> > > pursue once we have an agreed upon definition.
> > >
> > > In regards to that definition discussion. I feel comfortable with the
> > idea
> > > of the semantic meaning Dan raises though I can’t shake a slight
> > discomfort
> > > (maybe FOMO) with limiting values in the physical definition.
> > >
> > > A few naive questions:
> > > 1. Would a physical definition with NANs & Inf+/- allow efficient
> > > encodings like ALP?
> > > 2. If the physical representation allows values that make encodings
> like
> > > ALP impossible, can a tighter Logical type allow different encodings?
> > > 3. Can we RLE or binpack a vector?
> > >
> > > Best Regards,
> > > Kurtis C. Wright
> > > > On Oct 14, 2026, at 12:45, Daniel Weeks <dw...@apache.org> wrote:
> > > >
> > > > Hey Jan,
> > > >
> > > > There's a lot that I disagree with respect to the framing of these
> > > > arguments and how we should be thinking about introducing vector as a
> > > > logical type.  There's a repeated tendency to let the physical
> > > > representation or performance drive discussion around types as
> opposed
> > to
> > > > what it represents.
> > > >
> > > > In parquet, establishing semantic meaning is the purpose of logical
> > > types.
> > > > The logical type separates how to interpret a value from its physical
> > > > representation.  We can represent a timestamp with INT64 or even a
> > string
> > > > value, so why have a type for it? Even though a FIXED_SIZED_LIST and
> > > VECTOR
> > > > may have the exact same representation and performance, they have
> > > different
> > > > semantic meaning.  Databases and engines across the ecosystem already
> > > > recognize this differentiation and have separate representations.
> The
> > > > choice to separate these isn't because the discussion is too tedious;
> > > > they're fundamentally different things.
> > > >
> > > > The finite requirement is what makes a vector a reflection of what it
> > > > represents.  As we've discussed in the thread, non-finite values in
> the
> > > > representation violate the properties of a vector and the value
> ceases
> > to
> > > > be a vector.  On the other hand, a 0 vector is still a vector (you
> can
> > > > argue the utility of it, but it is a valid vector).  Implementations
> > can
> > > > choose what they validate or how to handle execution errors; that's
> an
> > > > implementation concern, not a requirement.  Operations performed on a
> > > > vector depend on the context in which they are used, but the
> > > representation
> > > > doesn't need to solve scenarios where that application doesn't make
> > > sense.
> > > >
> > > > The guarantees or context needed to use a vector do not need to be
> > > inherent
> > > > in the type for it to be a useful type.  We don't need the currency
> > used
> > > > for decimal or whether a floating point is a normalized value as part
> > of
> > > > the type to make use of those types.  This conflates the use cases
> with
> > > the
> > > > type system and unnecessarily attempts to draw the context into the
> > type
> > > > definition.  Similarly, you don't need to capture normalization,
> model,
> > > > etc. as part of the type for it to be useful.
> > > >
> > > > So the reason we want to add a vector type (or FINITE_VECTOR) is
> > because
> > > we
> > > > want to store vectors with the semantic meaning and requirements
> based
> > on
> > > > what they represent.
> > > >
> > > > -Dan
> > > >
> > > >
> > > >> On Wed, Oct 14, 2026 at 1:22 AM Jan Finis <jp...@gmail.com>
> wrote:
> > > >>
> > > >> A bit late to the party, but here we go:
> > > >>
> > > >> I have the strong sentiment that the FINITE_VECTOR type as it is
> > > currently
> > > >> being discussed will not be very useful and just increase
> > implementation
> > > >> complexity. The arguments for the type are all quite weak (details
> see
> > > >> below). The FINITE part doesn't seem to yield any value and is not a
> > > useful
> > > >> restriction by itself. The vector (i.e. fixed length) part should
> > > rather be
> > > >> generalized instead of making a one-off thing. In general, the
> > existing
> > > >> types are already working well for uncompressed vectors. The real
> > > challenge
> > > >> that would bring real benefits is supporting quantization. This is
> > where
> > > >> you get orders of magnitude improvements to storage space and
> > processing
> > > >> speed.
> > > >>
> > > >> Let's first discuss the FINITE part. So far, the only arguments I
> have
> > > read
> > > >> for this are:
> > > >>
> > > >> 1. many other vector databases have it
> > > >> 2. vectors are the hot stuff right now, so Parquet better boast
> having
> > > >> "native" type support for vectors
> > > >> 3. by focussing on this common special case, we can make rapid
> > progress,
> > > >> while a general solution would take too long to specify
> > > >> 4. to give a guarantee to readers that no INF/NaN/NULL will be in
> the
> > > >> vector
> > > >>
> > > >> the first two arguments are quite weak by themselves. Vector
> databases
> > > are
> > > >> very specialized engines. Such engines can of course be very
> > > opinionated on
> > > >> their data types. A general format like Parquet must instead weigh
> > > whether
> > > >> a special type is worth the added complexity over a more general,
> > widely
> > > >> applicable type. In this case, it is IMHO not at all. And just
> having
> > a
> > > >> type called "vector" for advertisement reasons to make Parquet
> appear
> > > more
> > > >> modern seems a non-goal in the first place.
> > > >>
> > > >> The third "time to market" argument is valid for systems that don't
> > > need a
> > > >> stable interface. It is a very dangerous attitude for formats like
> > > Parquet
> > > >> that must be stable long term and should have a consolidated,
> concise,
> > > >> non-redundant type system. For such long term storage formats, we
> need
> > > to
> > > >> take the time to build the right thing, as any shortcuts we take
> will
> > > stay
> > > >> as blemishes in the format forever. If I read the discussion
> > correctly,
> > > >> this is actually the main argument as of now: We have found that it
> is
> > > hard
> > > >> to align on how a fixed size list type should look like, so we want
> to
> > > go
> > > >> the easier route of a specialized type. This seems like a
> > > >> dangerous shortcut fueled by feature recency bias. Ask yourself:
> Would
> > > we
> > > >> want this, if Parquet already had a fixed size list type?
> > > >>
> > > >> So finally, the only technical argument is the fourth one: "It
> signals
> > > >> semantics and ensures that no bad data lands in the format". I see
> two
> > > big
> > > >> problems with this:
> > > >>
> > > >> 1. Readers *still* need to test for NaN / INF because Parquet files
> > may
> > > be
> > > >> corrupt. Nothing can stop a malicious or buggy writer from
> physically
> > > >> writing NaN or INF. *So you save no single line of code in a reader.
> > You
> > > >> save no branches, as you still need to handle the INF/NaN case.* So
> > this
> > > >> doesn't help with performance at all. The only difference is, that
> the
> > > >> error message changes from "found an invalid INF value in the data"
> to
> > > >> "Your parquet file seems to be corrupt". It is actually quite the
> > > opposite,
> > > >> readers and writers get more complex having to handle this new type.
> > > >>
> > > >> If you now argue that it's not about performance, it's about defined
> > > >> semantics and erroring out early in writers, then the second problem
> > > >> arises:
> > > >>
> > > >> 2. The proposed no-NaN/INF/NULL guarantee is just one of multiple
> > > >> guarantees that you need for sensible vector processing. If "no
> > > >> INF/NaN/NULL" was the only guarantee you needed, I would agree that
> > > >> FINITE_VECTOR is useful, but it is by no means the only guarantee
> you
> > > need,
> > > >> instead, depending on the distance metric, you also need the
> following
> > > >> guarantees. E.g.:
> > > >>
> > > >>   - No vector may be the 0 vector, otherwise, e.g., cosine
> similarity
> > is
> > > >>   undefined. Cosine similarity is often used as default, so this is
> > > >> actually
> > > >>   quite important in practice.
> > > >>   - If we want to use a dot product, then all vectors need to
> > guarantee
> > > a
> > > >>   length of 1. This would actually be quite useful for performance
> > > >> reasons.
> > > >>   - The vectors must come from the same model as the vector used for
> > the
> > > >>   search. Otherwise, the search yields a numeric result, but that
> > result
> > > >> has
> > > >>   no sensible meaning.
> > > >>
> > > >> So, what use is a type that just guarantees part of the things you
> > need
> > > for
> > > >> vector search but not others? You still need additional guarantees
> in
> > > >> addition to that type. What makes INF/NaN so important in comparison
> > to,
> > > >> say, "no 0 vector present", that it warrants an own type, while the
> > > other
> > > >> guarantees do not? If we really want this type as a basis for
> > > cross-engine
> > > >> sharing of vectors, then it needs to give *all* guarantees.
> > > >>
> > > >> I'm not against a specialized vector type if it really carries *all*
> > > >> information to be useful by itself. If it carried all that
> > information,
> > > >> then a reading engine could indeed validate that only meaningful
> > vector
> > > >> searches are executed. But as long as it is just a half-baked thing,
> > it
> > > is
> > > >> not useful by itself and just a burden on each implementer and
> > > maintainer.
> > > >> If we need extra information anyway, why not just stick to a general
> > > type
> > > >> like VECTOR (without FINITE).
> > > >>
> > > >> If you're a proponent of FINITE_VECTOR, ask yourself (or tell us)
> what
> > > >> advantage you will *really* get over just a (potentially fixed size)
> > > List
> > > >> of Float? As discussed, it's not performance. It's also not a
> semantic
> > > >> marker that is sufficient by itself. It's just a small hint that
> > allows
> > > >> writers to error out once they see a NaN or INF in the input, while
> > they
> > > >> will still happily write a bogus 0 vector. This seems to be of zero
> > > >> benefit.
> > > >>
> > > >> Anecdotally, we currently handle petabytes of vectors in production
> in
> > > >> Data360. We simply store them in a List of Float, with an optimized
> > > R-Level
> > > >> reader implementation that rapidly checks that all vectors are the
> > same
> > > >> length. This solution works just fine. The overhead of R-level
> > decoding
> > > is
> > > >> barely measurable. The check is super fast and R-levels are tiny in
> > > >> comparison to the payload vectors, so also the added I/O is
> > negligible.
> > > >> This is also why I'm so firm in my sentiment: We are running a
> > > production
> > > >> grade system processing trillions of vectors right now and see no
> big
> > > >> issues with the types that Parquet currently provides.
> > > >>
> > > >> As I don't just want to be a naysayer and point out problems with
> the
> > > >> current design, here are the directions that I would propose indeed:
> > > >>
> > > >>   - a general fixed size list type. Don't do shortcuts just because
> > the
> > > >>   discussion is too tedious.
> > > >>   - a vector type that indeed carries enough information to be
> really
> > > >>   useful for vector search by itself. (e.g., model name / version,
> > > >> optional
> > > >>   guarantees for no 0 vector / normalized vector length). With such
> a
> > > >> type,
> > > >>   you could really do well defined vector searches even if the
> writing
> > > >> engine
> > > >>   is completely different to the reading one.
> > > >>   - Standardize quantizations. This will be a real game changer for
> > > >>   performance.
> > > >>
> > > >> Cheers,
> > > >> Jan
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>> Am Mi., 14. Okt. 2026 um 03:36 Uhr schrieb Gang Wu <
> ustcwg@gmail.com
> > >:
> > > >>>
> > > >>> I think separating them into FINITE_VECTOR and FIXED_LENGTH_LIST
> is a
> > > >> good
> > > >>> idea, though I think renaming FINITE_VECTOR to VECTOR looks more
> > > natural.
> > > >>> Should we move forward with proposing FINITE_VECTOR/VECTOR first?
> > > >>>
> > > >>> On Tue, Oct 13, 2026 at 6:26 PM Alkis Evlogimenos via dev <
> > > >>> dev@parquet.apache.org> wrote:
> > > >>>
> > > >>>> I suggest we go with VECTOR for the finite vector to keep the name
> > the
> > > >>> same
> > > >>>> as other engines/type systems. If we want a non-finite one in the
> > > >> future
> > > >>> we
> > > >>>> can name that something else or we add fixed size list that will
> > serve
> > > >>> the
> > > >>>> same purpose without a special name.
> > > >>>>
> > > >>>>
> > > >>>> On Thu, Oct 8, 2026 at 10:20 PM Ryan Blue <bl...@apache.org>
> wrote:
> > > >>>>
> > > >>>>> A logical type would:
> > > >>>>> - Describe data for users and readers to understand what the file
> > > >>>> contains
> > > >>>>> - Signal that repetition and definition levels will reflect a
> fixed
> > > >>>> number
> > > >>>>> of elements, which implementations can use to optimize decoding
> > > >>>>> - Guarantee that there are no null elements (because the type
> > cannot
> > > >> be
> > > >>>>> applied to a list with optional elements)
> > > >>>>> - [FINITE_VECTOR] Guarantee that no elements are NaN or Infinity
> > > >> values
> > > >>>> so
> > > >>>>> that the vectors have a finite length and are suitable for
> distance
> > > >>>>> calculations
> > > >>>>>
> > > >>>>> The finite value restriction is a restriction on writers to
> prevent
> > > >> bad
> > > >>>>> data from being written.
> > > >>>>>
> > > >>>>> Ignoring logical types that are not understood is a rule for
> > > >> readers. A
> > > >>>>> reader can handle vectors and pass it on to another layer. For
> > > >>> instance,
> > > >>>> a
> > > >>>>> Parquet reader that doesn't support geometry passes values as
> > > >> `binary`.
> > > >>>>> That binary value can be passed to a geometry UDF, even if the
> > > >>> underlying
> > > >>>>> engine and Parquet reader doesn't have support.
> > > >>>>>
> > > >>>>> A tool that doesn't understand FINITE_VECTOR can similarly pass a
> > > >> list
> > > >>> of
> > > >>>>> floats along to a UDF. But it is still important to the user that
> > bad
> > > >>>>> values are not present in the data. In your example, bad values
> are
> > > >>>> vectors
> > > >>>>> that cause the centroid calculation to fail, like vectors with
> > > >> infinite
> > > >>>>> length.
> > > >>>>>
> > > >>>>> Ryan
> > > >>>>>
> > > >>>>> On Thu, Oct 8, 2026 at 12:54 PM Will Edwards via dev <
> > > >>>>> dev@parquet.apache.org>
> > > >>>>> wrote:
> > > >>>>>
> > > >>>>>> Could we clarify the use cases for a VECTOR annotation on
> > > >>> LIST<FLOAT>?
> > > >>>>>> Specifically, what does it make possible that's impossible
> today,
> > > >> and
> > > >>>>> what
> > > >>>>>> does it make easier that's currently hard?
> > > >>>>>>
> > > >>>>>> I'm struggling to find a strong answer. I'd like every engine to
> > > >>> offer
> > > >>>>>> vector functions on lists of floats without needing a new
> logical
> > > >>> type.
> > > >>>>>>
> > > >>>>>> To be clear, I'm still enthusiastic about a vector type that
> > > >>> describes
> > > >>>>> how
> > > >>>>>> to unpack and scale composite representations (quantised codes
> > plus
> > > >>>>> scales,
> > > >>>>>> codebooks, etc.), in the direction ML is going. That adds
> > > >> information
> > > >>>>>> engines don't have. A label on LIST<FLOAT> doesn't: engines
> > already
> > > >>>>>> understand the physical type, and many already offer vector
> > > >> functions
> > > >>>> on
> > > >>>>>> it. I don't see a performance benefit, and NaN/null/Inf checks
> are
> > > >>>> cheap
> > > >>>>>> enough that engines should do them anyway. The label also
> doesn't
> > > >> say
> > > >>>>> what
> > > >>>>>> you can do with the column. Knowing that a column holds
> embeddings
> > > >>>>> doesn't
> > > >>>>>> say how they can be used, just as zipcode and age are both
> > integers
> > > >>> but
> > > >>>>>> multiplying them is meaningless.
> > > >>>>>>
> > > >>>>>> My concern is the opposite effect. We're establishing that
> engines
> > > >>>> should
> > > >>>>>> ignore logical types they don't understand. So a tool that
> doesn't
> > > >>> know
> > > >>>>>> VECTOR will read the column as LIST<FLOAT>, let the user compute
> > > >>>>> centroids
> > > >>>>>> (say), and write plain LIST<FLOAT> back out. If other engines
> > > >>> restrict
> > > >>>>>> embedding functions to VECTOR-annotated columns, that file stops
> > > >>>> working
> > > >>>>>> with them, through no fault of the user. An annotation that
> > doesn't
> > > >>>>> change
> > > >>>>>> what the data is shouldn't be able to make it unusable.
> > > >>>>>>
> > > >>>>>> Am I missing a use case?
> > > >>>>>>
> > > >>>>>> Best, Will
> > > >>>>>>
> > > >>>>>> On Thu, 8 Oct 2026 at 19:53, Steven Wu <st...@gmail.com>
> > > >> wrote:
> > > >>>>>>
> > > >>>>>>> I also think a FINITE_VECTOR type brings a lot of values to
> many
> > > >>> use
> > > >>>>>> cases.
> > > >>>>>>> It is would be great to make progress on the FINITE_VECTOR, and
> > > >> not
> > > >>>> to
> > > >>>>>> fit
> > > >>>>>>> every use cases into the FIXED_LENGTH_LIST.
> > > >>>>>>>
> > > >>>>>>> On Mon, Oct 5, 2026 at 1:26 PM Ryan Blue <bl...@apache.org>
> > > >> wrote:
> > > >>>>>>>
> > > >>>>>>>> Going through this thread again, it looks like there are a few
> > > >>>> things
> > > >>>>>>> that
> > > >>>>>>>> people generally agree on:
> > > >>>>>>>>
> > > >>>>>>>> - In the short term, we want to introduce a logical type or
> > > >>> types;
> > > >>>> a
> > > >>>>>>> better
> > > >>>>>>>> physical representation can be done later. We can focus on
> > > >>> logical
> > > >>>>>> types.
> > > >>>>>>>> - There is precedent in logical types for restricting values,
> > > >>> like
> > > >>>>> UTF8
> > > >>>>>>>> strings to contain only UTF-8 codepoints. I think we also
> agree
> > > >>>> that
> > > >>>>> we
> > > >>>>>>>> could restrict vector elements to non-nested or primitive
> > > >> types.
> > > >>>>>>>> - There are many examples of databases that restrict their
> > > >> vector
> > > >>>>> types
> > > >>>>>>> to
> > > >>>>>>>> non-null, finite elements.
> > > >>>>>>>> - Some people want to have write-time guarantees that vectors
> > > >>> have
> > > >>>>> only
> > > >>>>>>>> non-null, finite elements
> > > >>>>>>>> - Some people want to extend this work to include tensors,
> > > >>> shorter
> > > >>>>>>> floating
> > > >>>>>>>> point representations, variable lengths, NaN or Infinity
> > > >>> elements,
> > > >>>>> etc.
> > > >>>>>>>>
> > > >>>>>>>> It sounds like we are nearly there, but have two halves of the
> > > >>>>>> community
> > > >>>>>>>> that care about different things. That just means we should
> > > >> have
> > > >>>>>> multiple
> > > >>>>>>>> types. It's entirely reasonable to want a type that allows NaN
> > > >>>>> values,
> > > >>>>>>> and
> > > >>>>>>>> it's also reasonable for those of us building databases, table
> > > >>>>>> standards,
> > > >>>>>>>> and vector indexes to care about guarantees that elements are
> > > >>>> finite.
> > > >>>>>>>> There's more than enough examples to show that this is a
> > > >>> legitimate
> > > >>>>> use
> > > >>>>>>>> case.
> > > >>>>>>>>
> > > >>>>>>>> Since I don't see the concerns of the different use cases
> > > >>> changing,
> > > >>>>> it
> > > >>>>>>>> looks like we need to find a compromise to move forward. I
> > > >>> propose
> > > >>>>> that
> > > >>>>>>> we
> > > >>>>>>>> introduce a FINITE_VECTOR type and separately a
> > > >> FIXED_LENGTH_LIST
> > > >>>> or
> > > >>>>>>>> another VECTOR type that allows infinite and NaN values.
> > > >>>>>>>>
> > > >>>>>>>> Can we agree on a compromise? Is there a strong reason to try
> > > >> to
> > > >>>> fit
> > > >>>>>>> these
> > > >>>>>>>> use cases into a single type? What if that means not making
> > > >>>> progress?
> > > >>>>>>>>
> > > >>>>>>>> Ryan
> > > >>>>>>>>
> > > >>>>>>>> On Mon, Oct 5, 2026 at 12:15 PM Philipp Fischbeck via dev <
> > > >>>>>>>> dev@parquet.apache.org> wrote:
> > > >>>>>>>>
> > > >>>>>>>>> Thinking about this more, I am also fine with pulling out the
> > > >>>>>> “NaN/Inf”
> > > >>>>>>>>> requirement. Either into a type boolean (my preference), or
> > > >>>> putting
> > > >>>>>> no
> > > >>>>>>>>> restrictions on the Parquet level, allowing all IEEE 754
> > > >> float
> > > >>>>>> numbers.
> > > >>>>>>>> If
> > > >>>>>>>>> some top-level application wants to put additional
> > > >> constraints
> > > >>> on
> > > >>>>> the
> > > >>>>>>>> data
> > > >>>>>>>>> type, like Iceberg not allowing NaN/infinity, it can still do
> > > >>> so
> > > >>>>> one
> > > >>>>>>>> layer
> > > >>>>>>>>> above. It seems like this was the biggest point of discussion
> > > >>>>>>> remaining.
> > > >>>>>>>>> All these options work for me, and we should move forward
> > > >> with
> > > >>>> this
> > > >>>>>>>>> proposal.
> > > >>>>>>>>>
> > > >>>>>>>>> Concerning additional element types like bf16, they can be
> > > >>>>> introduced
> > > >>>>>>> as
> > > >>>>>>>>> Parquet types (at least in the current LIST-based spec
> > > >>> proposal).
> > > >>>>>>>>>
> > > >>>>>>>>> Concerning Will’s suggestion around representing VECTOR as a
> > > >>>>> variable
> > > >>>>>>>>> length list with a MIN/MAX_LENGTH metadata restriction: I
> > > >> think
> > > >>>>> that
> > > >>>>>> a
> > > >>>>>>>>> dedicated logical VECTOR type still provides enough benefits
> > > >> by
> > > >>>>>>>> describing
> > > >>>>>>>>> the semantics of the data, and allowing to capture future
> > > >>> changes
> > > >>>>>>> around
> > > >>>>>>>>> statistics. In that case, the length is better captured as an
> > > >>>> exact
> > > >>>>>>>> length
> > > >>>>>>>>> for this type specifically. I’m also not sure about other use
> > > >>>> cases
> > > >>>>>> for
> > > >>>>>>>>> MIN_LENGTH, so introducing a new type constraint system just
> > > >>> for
> > > >>>>> the
> > > >>>>>>>>> MIN_LENGTH=MAX_LENGTH case for vectors doesn’t have enough
> > > >>> value
> > > >>>> in
> > > >>>>>> my
> > > >>>>>>>>> view.
> > > >>>>>>>>>
> > > >>>>>>>>> Best,
> > > >>>>>>>>> Philipp
> > > >>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>> On Thu, Oct 1, 2026 at 9:00 AM Will Edwards via dev <
> > > >>>>>>>>> dev@parquet.apache.org>
> > > >>>>>>>>> wrote:
> > > >>>>>>>>>
> > > >>>>>>>>>> +1 Sebastian, a vector type suggests usage beyond
> > > >> normalized
> > > >>>>>>> embeddings
> > > >>>>>>>>> and
> > > >>>>>>>>>> engines really aren't going to be able to elide all the
> > > >>> checks
> > > >>>>>>> anyway.
> > > >>>>>>>>>>
> > > >>>>>>>>>> Antoine, you suggested we explore metadata solutions too :)
> > > >>>>>>>>>>
> > > >>>>>>>>>> Today, a database supporting CHAR(N) or VARCHAR(n) can't
> > > >>>> express
> > > >>>>>> this
> > > >>>>>>>> in
> > > >>>>>>>>>> Parquet.  Engines can write opaque blobs they understand in
> > > >>> the
> > > >>>>>>>> metadata
> > > >>>>>>>>>> (e.g. Arrow does) so the extra type info round-trips, but
> > > >>> this
> > > >>>>>> isn't
> > > >>>>>>>>>> standardized, portable, nor interoperable.
> > > >>>>>>>>>>
> > > >>>>>>>>>> So perhaps standardizing constraint metadata or new Thrift
> > > >>>> fields
> > > >>>>>> (my
> > > >>>>>>>>> pref)
> > > >>>>>>>>>> can generically support VECTOR(N) whilst also being more
> > > >>>> broadly
> > > >>>>>>> useful
> > > >>>>>>>>> for
> > > >>>>>>>>>> other use cases and data types?
> > > >>>>>>>>>>
> > > >>>>>>>>>> Imagine metadata or Thrift fields to tag the MIN_LENGTH and
> > > >>>>>>> MAX_LENGTH
> > > >>>>>>>>> for
> > > >>>>>>>>>> Parquet's variable-length physcial types.  These would
> > > >> count
> > > >>>>>> logical
> > > >>>>>>>>> units
> > > >>>>>>>>>> (e.g. UTF8 code points for STRING) rather than bytes.
> > > >>>>>>>>>>
> > > >>>>>>>>>> Then one engine's VECTOR(N) can be written to Parquet as
> > > >>>> LIST(N)
> > > >>>>>> with
> > > >>>>>>>>>> MIN_LENGTH = MAX_LENGTH = N and then readers - even
> > > >> different
> > > >>>>>>> engines -
> > > >>>>>>>>> can
> > > >>>>>>>>>> pick up these constraints if they support doing so.  We
> > > >> don't
> > > >>>>>>> actually
> > > >>>>>>>>> need
> > > >>>>>>>>>> a logical type to express a VECTOR then.
> > > >>>>>>>>>>
> > > >>>>>>>>>> This also allows standard SQL's CHAR(N) and VARCHAR(N) to
> > > >> be
> > > >>>>>>> expressed
> > > >>>>>>>> on
> > > >>>>>>>>>> UTF8 columns etc.
> > > >>>>>>>>>>
> > > >>>>>>>>>> This also seems generally useful in the future as we
> > > >> approach
> > > >>>>>>>> expressing
> > > >>>>>>>>>> other tensor data shapes and types.
> > > >>>>>>>>>>
> > > >>>>>>>>>> This won't change the mechanics: the storage type doesn't
> > > >>>> enforce
> > > >>>>>>> these
> > > >>>>>>>>>> constraints and engines must check them. It's previously
> > > >>>>>> established
> > > >>>>>>>> that
> > > >>>>>>>>>> performance is just a software problem and there is no
> > > >>>>> fundamental
> > > >>>>>>>>>> performance advantage to the fixed size types in
> > > >> Dremel-like
> > > >>>>>> formats
> > > >>>>>>>> like
> > > >>>>>>>>>> Parquet.
> > > >>>>>>>>>>
> > > >>>>>>>>>> Best,
> > > >>>>>>>>>> Will
> > > >>>>>>>>>>
> > > >>>>>>>>>> On Wed, 30 Sept 2026 at 16:08, Sebastian Baunsgaard <
> > > >>>>>>>>> baunsgaard@apache.org
> > > >>>>>>>>>>>
> > > >>>>>>>>>> wrote:
> > > >>>>>>>>>>
> > > >>>>>>>>>>> After listening in on the last meeting Wednesday, I have
> > > >>> been
> > > >>>>>>>> thinking
> > > >>>>>>>>>>> about this quite a bit, and thought I would share my 2
> > > >>> cents.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> I suggest not requiring VECTOR elements to be finite, as
> > > >>>>> proposed
> > > >>>>>>> in
> > > >>>>>>>>> the
> > > >>>>>>>>>> PR
> > > >>>>>>>>>>> <https://github.com/apache/parquet-format/pull/624>.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> While Divjot and others note the specialization for
> > > >> UTF-8,
> > > >>>>>> Parquet
> > > >>>>>>>>>> already
> > > >>>>>>>>>>> defines DOUBLE and FLOAT as IEEE encodings
> > > >>>>>>>>>>> <
> > > >>>>>>>>
> > > >>>>
> > https://github.com/apache/parquet-format/blob/master/README.md#types
> > > >>>>>>>>>> .
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Making NaN or +/-Inf illegal inside VECTOR would be a new
> > > >>>>>>> restriction
> > > >>>>>>>>> on
> > > >>>>>>>>>>> those same underlying physical types. Scalar FLOAT/DOUBLE
> > > >>>>> columns
> > > >>>>>>>> have
> > > >>>>>>>>>>> never had such a rule.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> It would also not match how the systems that actually
> > > >>> produce
> > > >>>>>> these
> > > >>>>>>>>>> vectors
> > > >>>>>>>>>>> treat floats. Most of them use IEEE floats, e.g. Spark,
> > > >>>> DuckDB,
> > > >>>>>>>> NumPy,
> > > >>>>>>>>>>> PyTorch, TensorFlow, or lower level CUDA, oneMKL, and
> > > >>>> OpenBLAS.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> pgvector's "all elements must be finite" rule is an
> > > >> insert
> > > >>>>>> policy.
> > > >>>>>>>>> While
> > > >>>>>>>>>> I
> > > >>>>>>>>>>> think Parquet as a general-purpose format should allow
> > > >> them
> > > >>>> out
> > > >>>>>> of
> > > >>>>>>>> the
> > > >>>>>>>>>> box.
> > > >>>>>>>>>>> It is a user system decision to disallow writing these
> > > >>> values
> > > >>>>> or
> > > >>>>>> to
> > > >>>>>>>>> throw
> > > >>>>>>>>>>> errors when encountering them.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> A type-level ban also does not keep Inf/NaN out of vector
> > > >>>> math.
> > > >>>>>>>> Finite
> > > >>>>>>>>>>> coordinates still overflow. For instance: magnitude =
> > > >>>>>> ||[FLOAT_MAX,
> > > >>>>>>>>>>> FLOAT_MAX]|| where every coordinate is finite overflows
> > > >> to
> > > >>>>> +Inf.
> > > >>>>>>>>> PyTorch
> > > >>>>>>>>>>> documents the same overflow: tensor([1e20, 1e20]).norm()
> > > >> is
> > > >>>> inf
> > > >>>>>> in
> > > >>>>>>>>> fp32.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> A generated MAGNITUDE(vec) column would then be Inf while
> > > >>> the
> > > >>>>>>> VECTOR
> > > >>>>>>>>>> itself
> > > >>>>>>>>>>> was required to be finite. The ban only forbids storing
> > > >>>>> specials
> > > >>>>>> in
> > > >>>>>>>> the
> > > >>>>>>>>>>> coordinates, and does not prevent even simple downstream
> > > >>>>>>> computations
> > > >>>>>>>>>> from
> > > >>>>>>>>>>> producing them.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> I suggest keeping VECTOR as the type and add an optional
> > > >>> bool
> > > >>>>>>>>>>> VectorType.finite (default false). Engines that want
> > > >>>>>> pgvector-like
> > > >>>>>>>>>>> rejection can require the flag = true. Engines that use
> > > >>>> NumPy /
> > > >>>>>>>>> PyTorch /
> > > >>>>>>>>>>> Spark arrays can keep representable IEEE values. For the
> > > >>> same
> > > >>>>>>>> reason, I
> > > >>>>>>>>>> do
> > > >>>>>>>>>>> not favor adding yet another type called FINITE_VECTOR.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Best,
> > > >>>>>>>>>>> Sebastian
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> On Tue, Sep 29, 2026 at 2:03 PM Divjot Arora via dev <
> > > >>>>>>>>>>> dev@parquet.apache.org>
> > > >>>>>>>>>>> wrote:
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>> I agree with moving forward with the proposal and
> > > >>> renaming
> > > >>>>>> VECTOR
> > > >>>>>>>> to
> > > >>>>>>>>>>>> FINITE_VECTOR or something
> > > >>>>>>>>>>>> similar if others feel that better communicates its
> > > >>>> intent. I
> > > >>>>>> see
> > > >>>>>>>>> value
> > > >>>>>>>>>>> in
> > > >>>>>>>>>>>> having a vector type that guarantees
> > > >>>>>>>>>>>> its values are mathematically-correct vectors for a few
> > > >>>>>> reasons:
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> 1. As others have mentioned, this is similar to the
> > > >>> STRING
> > > >>>>>>> logical
> > > >>>>>>>>> type
> > > >>>>>>>>>>>> mandating UTF-8.
> > > >>>>>>>>>>>> 2. It is not possible to rely on statistics for
> > > >> detecting
> > > >>>> NaN
> > > >>>>>> or
> > > >>>>>>>>> +-inf
> > > >>>>>>>>>>>> values as all statistics are optional.
> > > >>>>>>>>>>>> To my knowledge, there is no precedent for requiring
> > > >> that
> > > >>>>>> writers
> > > >>>>>>>>> emit
> > > >>>>>>>>>>>> stats for specific logical types.
> > > >>>>>>>>>>>> 3. Without statistics, NaN/inf values could flow
> > > >> through
> > > >>>> the
> > > >>>>>>> engine
> > > >>>>>>>>> and
> > > >>>>>>>>>>>> block them for specific functionality.
> > > >>>>>>>>>>>> This seems to be Lance's behavior. However, I don't
> > > >> think
> > > >>>>> this
> > > >>>>>> is
> > > >>>>>>>> as
> > > >>>>>>>>>> easy
> > > >>>>>>>>>>>> as it sounds and Lance has run
> > > >>>>>>>>>>>> into several bugs due to this ([1] [2] for examples).
> > > >>> Their
> > > >>>>>>> Python
> > > >>>>>>>>> [3]
> > > >>>>>>>>>>> and
> > > >>>>>>>>>>>> Rust [4] APIs default to erroring on
> > > >>>>>>>>>>>> vectors that contain NaN values.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> I agree that Parquet is a general-purpose format, but I
> > > >>>> don't
> > > >>>>>>> think
> > > >>>>>>>>>> this
> > > >>>>>>>>>>>> implies the lack of domain-specific types.
> > > >>>>>>>>>>>> Instead, this broadens the use cases where Parquet
> > > >> would
> > > >>> be
> > > >>>>>>>>> considered
> > > >>>>>>>>>> a
> > > >>>>>>>>>>>> good fit. I also fully agree with
> > > >>>>>>>>>>>> continuing work on a general fixed-size list type if we
> > > >>>>> believe
> > > >>>>>>>> there
> > > >>>>>>>>>> are
> > > >>>>>>>>>>>> non-vector use cases that would benefit
> > > >>>>>>>>>>>> from such guarantees at the storage layer.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> [1] https://github.com/lance-format/lance/pull/3648
> > > >>>>>>>>>>>> [2] https://github.com/lance-format/lance/issues/8607
> > > >>> (see
> > > >>>>>>> point 2
> > > >>>>>>>>> in
> > > >>>>>>>>>>>> "root
> > > >>>>>>>>>>>> cause")
> > > >>>>>>>>>>>> [3]
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>
> > > >>>>>>
> > > >>>>>
> > > >>>>
> > > >>>
> > > >>
> > >
> >
> https://github.com/lancedb/lancedb/blob/0327618a62483feb6dca2336549845e5887cd48f/python/python/lancedb/table.py#L4807-L4911
> > > >>>>>>>>>>>> [4]
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>
> > > >>>>>>
> > > >>>>>
> > > >>>>
> > > >>>
> > > >>
> > >
> >
> https://github.com/lancedb/lancedb/blob/0327618a62483feb6dca2336549845e5887cd48f/rust/lancedb/src/table/add_data.rs#L105-L110
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> On Mon, Sep 28, 2026 at 6:19 PM Philipp Fischbeck via
> > > >>> dev <
> > > >>>>>>>>>>>> dev@parquet.apache.org> wrote:
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>> I also favor moving forward with the proposal as is.
> > > >> I
> > > >>>> see
> > > >>>>>>> strong
> > > >>>>>>>>>> value
> > > >>>>>>>>>>>> and
> > > >>>>>>>>>>>>> use cases for the numeric vector type, and I agree
> > > >> with
> > > >>>> the
> > > >>>>>>>>>>> restrictions
> > > >>>>>>>>>>>>> around NaN/Inf values and element types in general.
> > > >>>>> Precedent
> > > >>>>>>>>> exists
> > > >>>>>>>>>> in
> > > >>>>>>>>>>>>> other types that restrict which physical type values
> > > >>> are
> > > >>>>>>>> logically
> > > >>>>>>>>>>> valid
> > > >>>>>>>>>>>>> (e.g., the geospatial types needing to be
> > > >> WKB-encoded).
> > > >>>> A
> > > >>>>>>>> general
> > > >>>>>>>>>>>>> fixed-size list also has value, but I think it makes
> > > >>>> sense
> > > >>>>> to
> > > >>>>>>>>>> separate
> > > >>>>>>>>>>>> the
> > > >>>>>>>>>>>>> two types. The semantics are relevant, and I can see
> > > >>> how
> > > >>>>> the
> > > >>>>>>> two
> > > >>>>>>>>>> types
> > > >>>>>>>>>>>>> might diverge further in the future regarding
> > > >>> statistics
> > > >>>> or
> > > >>>>>>>>>> encodings.
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> I would be okay with renaming the vector type, but it
> > > >>>> seems
> > > >>>>>>> like
> > > >>>>>>>>> the
> > > >>>>>>>>>>>> VECTOR
> > > >>>>>>>>>>>>> name is already used across other engines mentioned
> > > >> in
> > > >>>> this
> > > >>>>>>>> thread.
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> Best,
> > > >>>>>>>>>>>>> Philipp
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> On Sat, Sep 26, 2026 at 4:37 PM Gang Wu <
> > > >>>> ustcwg@gmail.com>
> > > >>>>>>>> wrote:
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> To separate orthogonal concerns, I feel we should
> > > >>> treat
> > > >>>>> the
> > > >>>>>>>>> general
> > > >>>>>>>>>>>>>> FixedSizeList as the underlying physical layout,
> > > >> and
> > > >>>>>> separate
> > > >>>>>>>> it
> > > >>>>>>>>>> from
> > > >>>>>>>>>>>> the
> > > >>>>>>>>>>>>>> discussion of a specialized VECTOR (or broader ML
> > > >>>> tensor)
> > > >>>>>>>> logical
> > > >>>>>>>>>>> type.
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> Currently, the immediate need is to define a
> > > >>>> specialized
> > > >>>>>>> VECTOR
> > > >>>>>>>>>> type
> > > >>>>>>>>>>> so
> > > >>>>>>>>>>>>>> that different engines and formats have a common
> > > >>>> semantic
> > > >>>>>>>>> contract
> > > >>>>>>>>>>> for
> > > >>>>>>>>>>>>>> interoperability.
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> If the main point of contention is just the
> > > >>>> NaN/Inf/null
> > > >>>>>>>>>> limitations,
> > > >>>>>>>>>>>>> maybe
> > > >>>>>>>>>>>>>> we can resolve this by adding a qualification to
> > > >> the
> > > >>>> name
> > > >>>>>>> (like
> > > >>>>>>>>>>> Andrew
> > > >>>>>>>>>>>>>> suggested, e.g., VECTOR_EMBEDDING or FINITE_VECTOR)
> > > >>> to
> > > >>>>>> avoid
> > > >>>>>>>>>>> confusion
> > > >>>>>>>>>>>>> with
> > > >>>>>>>>>>>>>> broader ML use cases. Alternatively, we could
> > > >>>> introduce a
> > > >>>>>>>> boolean
> > > >>>>>>>>>>> flag
> > > >>>>>>>>>>>>>> (e.g., optional bool VectorType.finite) as proposed
> > > >>>>>> earlier.
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> On Fri, Sep 25, 2026 at 3:52 AM Andrew Lamb <
> > > >>>>>>>>>> andrewlamb11@gmail.com>
> > > >>>>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> I am a little confused about the points of
> > > >>>> disagreement
> > > >>>>>>> here.
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> Is it that the name `VECTOR` is too general for
> > > >> the
> > > >>>>>>> specific
> > > >>>>>>>>> use
> > > >>>>>>>>>>> case
> > > >>>>>>>>>>>>> of
> > > >>>>>>>>>>>>>>> fixed length arrays with non-finite values used
> > > >> for
> > > >>>>>> vector
> > > >>>>>>>>>>>> embeddings?
> > > >>>>>>>>>>>>>> If
> > > >>>>>>>>>>>>>>> so, maybe we could add a qualification to the
> > > >> name
> > > >>>>>>>>>> (FINITE_VECTOR?
> > > >>>>>>>>>>>>>>> FEATURE_VECTOR? VECTOR_EMBEDDING?)
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> It seems like there is precedent for defining
> > > >>> logical
> > > >>>>>> types
> > > >>>>>>>>> that
> > > >>>>>>>>>>>>> specify
> > > >>>>>>>>>>>>>>> certain physical types and restrict their
> > > >> contents
> > > >>> to
> > > >>>>>> help
> > > >>>>>>>>>> readers
> > > >>>>>>>>>>>>>>> interpret meaning, for example string[1] or JSON;
> > > >>>>>> Defining
> > > >>>>>>>> one
> > > >>>>>>>>>> for
> > > >>>>>>>>>>>>>> vector
> > > >>>>>>>>>>>>>>> embeddings seems reasonable enough to me.
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> Andrew
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> [1]:
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>
> > > >>>>>>
> > > >>>>>
> > > >>>>
> > > >>>
> > > >>
> > >
> >
> https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#string
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> On Thu, Sep 24, 2026 at 3:34 PM Daniel Weeks <
> > > >>>>>>>>> dweeks@apache.org>
> > > >>>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> Well, Parquet is not a single-purpose format.
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> Vector stores can choose whatever policy fits
> > > >>>> their
> > > >>>>>>>> usage,
> > > >>>>>>>>>> but
> > > >>>>>>>>>>>>> we're
> > > >>>>>>>>>>>>>>>>> talking about a general-purpose file format
> > > >>> that
> > > >>>>> aims
> > > >>>>>>> to
> > > >>>>>>>> be
> > > >>>>>>>>>>>> broadly
> > > >>>>>>>>>>>>>>>>> applicable.
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> I agree that the format is intended to be
> > > >> general
> > > >>>>>>> purpose,
> > > >>>>>>>>> but
> > > >>>>>>>>>>>> types
> > > >>>>>>>>>>>>>>> within
> > > >>>>>>>>>>>>>>>> the format define specific representation and
> > > >>>>> semantics
> > > >>>>>>> to
> > > >>>>>>>>>>> achieve
> > > >>>>>>>>>>>>> that
> > > >>>>>>>>>>>>>>>> goal.
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> "Vector" has a more general sense than ML and
> > > >>>> it's
> > > >>>>>>> quite
> > > >>>>>>>>>>> trivial
> > > >>>>>>>>>>>> to
> > > >>>>>>>>>>>>>>>>> produce vectors with non-finite values using
> > > >>> e.g.
> > > >>>>>> NumPy
> > > >>>>>>>> or
> > > >>>>>>>>>>> Pandas
> > > >>>>>>>>>>>>> (or
> > > >>>>>>>>>>>>>>>>> any other piece of dataframe or scientific
> > > >>>>> computing
> > > >>>>>>>>>> software).
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> The justification for restricting non-finite
> > > >>>> values
> > > >>>>> is
> > > >>>>>>>>>> directly
> > > >>>>>>>>>>>>>> related
> > > >>>>>>>>>>>>>>> to
> > > >>>>>>>>>>>>>>>> the semantic meaning of a vector.  Vectors have
> > > >>>>>> specific
> > > >>>>>>>>>>>>>> characteristics
> > > >>>>>>>>>>>>>>>> (dimension, magnitude, direction, etc).
> > > >>> Non-finite
> > > >>>>>>> values
> > > >>>>>>>>> are
> > > >>>>>>>>>>>>> invalid
> > > >>>>>>>>>>>>>>> and
> > > >>>>>>>>>>>>>>>> break those characteristics.  I'd liken it to
> > > >>>> storing
> > > >>>>>>>>> non-utf8
> > > >>>>>>>>>>>> values
> > > >>>>>>>>>>>>>> in
> > > >>>>>>>>>>>>>>> a
> > > >>>>>>>>>>>>>>>> string and still calling it a string.
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> That also differentiates a vector from a fixed
> > > >>>> length
> > > >>>>>>> list,
> > > >>>>>>>>>> which
> > > >>>>>>>>>>>> is
> > > >>>>>>>>>>>>>> more
> > > >>>>>>>>>>>>>>>> general and does not carry the same semantic
> > > >>>> meaning.
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> I also don't think we need to bundle all
> > > >>> similarly
> > > >>>>>> shaped
> > > >>>>>>>> use
> > > >>>>>>>>>>> cases
> > > >>>>>>>>>>>>>> into
> > > >>>>>>>>>>>>>>>> the same solution (e.g. flxed size list).
> > > >> Vector
> > > >>>> is
> > > >>>>> a
> > > >>>>>>> well
> > > >>>>>>>>>>>>> established
> > > >>>>>>>>>>>>>>>> data type at this point and aligns with the
> > > >>> general
> > > >>>>>>> purpose
> > > >>>>>>>>>>> nature
> > > >>>>>>>>>>>> of
> > > >>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>> format (many other formats and databases
> > > >> natively
> > > >>>>>> support
> > > >>>>>>>>>> this).
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> -Dan
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> On Thu, Sep 24, 2026 at 11:55 AM Antoine
> > > >> Pitrou <
> > > >>>>>>>>>>>> antoine@python.org>
> > > >>>>>>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> Le 24/09/2026 à 20:11, Russell Spitzer a
> > > >> écrit
> > > >>> :
> > > >>>>>>>>>>>>>>>>>> I think something I haven't seen yet is the
> > > >>>> clear
> > > >>>>>>>>> use-case
> > > >>>>>>>>>>> for
> > > >>>>>>>>>>>>>>>> non-finite
> > > >>>>>>>>>>>>>>>>>> values within a vector type. There has
> > > >> been a
> > > >>>> lot
> > > >>>>>> of
> > > >>>>>>>>>>>> theoretical
> > > >>>>>>>>>>>>>>>>> discussion
> > > >>>>>>>>>>>>>>>>>> here about use cases but what toolings
> > > >>>> currently
> > > >>>>>>>> produce
> > > >>>>>>>>>>> these
> > > >>>>>>>>>>>>>>> vectors
> > > >>>>>>>>>>>>>>>>> and
> > > >>>>>>>>>>>>>>>>>> how do they function in downstream engines?
> > > >>> I'd
> > > >>>>>> like
> > > >>>>>>> to
> > > >>>>>>>>>> limit
> > > >>>>>>>>>>>>>>>> introducing
> > > >>>>>>>>>>>>>>>>>> types that have a wide range of values
> > > >> which
> > > >>>> are
> > > >>>>>>> simply
> > > >>>>>>>>> are
> > > >>>>>>>>>>>>> failure
> > > >>>>>>>>>>>>>>>> modes
> > > >>>>>>>>>>>>>>>>>> of different flavors
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> "Vector" has a more general sense than ML and
> > > >>>> it's
> > > >>>>>>> quite
> > > >>>>>>>>>>> trivial
> > > >>>>>>>>>>>> to
> > > >>>>>>>>>>>>>>>>> produce vectors with non-finite values using
> > > >>> e.g.
> > > >>>>>> NumPy
> > > >>>>>>>> or
> > > >>>>>>>>>>> Pandas
> > > >>>>>>>>>>>>> (or
> > > >>>>>>>>>>>>>>>>> any other piece of dataframe or scientific
> > > >>>>> computing
> > > >>>>>>>>>> software).
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> It's not like the general concept of a vector
> > > >>> is
> > > >>>> a
> > > >>>>>> IETF
> > > >>>>>>>>>>> standard
> > > >>>>>>>>>>>> or
> > > >>>>>>>>>>>>>>>>> anything, so analogies with JSON or UTF8
> > > >> (where
> > > >>>> the
> > > >>>>>>>> values
> > > >>>>>>>>>> are
> > > >>>>>>>>>>>>>> expected
> > > >>>>>>>>>>>>>>>>> to be valid according to the respective
> > > >> specs)
> > > >>> do
> > > >>>>> not
> > > >>>>>>>>> apply.
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> This discussion would IMHO have a better
> > > >> chance
> > > >>>> of
> > > >>>>>>>>> converging
> > > >>>>>>>>>>> if
> > > >>>>>>>>>>>> we
> > > >>>>>>>>>>>>>>>>> separated the two orthogonal concerns of:
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> 1. representing fixed-length list/vector data
> > > >>> in
> > > >>>>>>> Parquet
> > > >>>>>>>>>> (with
> > > >>>>>>>>>>> a
> > > >>>>>>>>>>>>>>>>> potential limitation to non-nested types)
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> 2. representing the information that a given
> > > >>>> column
> > > >>>>>> (or
> > > >>>>>>>>>> column
> > > >>>>>>>>>>>>> chunk,
> > > >>>>>>>>>>>>>>>>> etc.) does not contain non-finite values
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> ... both of which are useful on their own
> > > >>> right,
> > > >>>>> but
> > > >>>>>> do
> > > >>>>>>>> not
> > > >>>>>>>>>>> have
> > > >>>>>>>>>>>> to
> > > >>>>>>>>>>>>>> be
> > > >>>>>>>>>>>>>>>>> conflated and bundled together.
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> Regards
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> Antoine.
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>
> > > >>>>>>
> > > >>>>>
> > > >>>>
> > > >>>
> > > >>
> > >
> > >
> >
>

Re: [DISCUSS] Numeric Vector type: physical representation (outcome of focused call)

[Ryan Blue <bl...@apache.org>]

I agree with much of what Dan said, but I'll add some to this thread as
well.

First, I want to address the arguments that were pointed out:
1. Other vector databases have finiteness requirements
2. "Vectors are the hot stuff right now" -- let's jump on the bandwagon
3. A general solution will take too long, finiteness will make faster
progress
4. To give a guarantee to readers that no INF/NaN/NULL will be in the vector

I don't think these represent our actual motivation or position, which
makes them function as a straw-man. This is probably not what was
intended, but I want to make sure that the misunderstanding
doesn't continue. For one thing, we aren't saying that we need finite
vectors because they are "hot stuff".

We want to build tools that use them on top of Parquet. I think it should
also be clear that we aren't advocating for finite elements blindly just
because other engines do (cargo cult design!). There are real
interoperability issues if we choose to go a different direction so this is
a "Chesterton's Fence" situation. It's important to understand why
those databases
made that choice. I've also pointed this out as confirmation that we're
going down a reasonable and established path.

The third argument sounds like a simple misunderstanding. We're pursuing a
logical type right now because that's forward compatible and we want to
label the data properly and set expectations for the type. Physical type
changes break forward compatibility and need to be released in a new format
version . I think this is the source of the confusion since it has been in
the same discussion, but the choice about finiteness is separate.

That brings us to the last argument, whether there is value in writers
participating in guarantees. I think there's value in adding requirements
for writers. It's up to readers how they want to use those requirements,
but a writer requirement allows a policy of "garbage in garbage out" rather
than being forced to handle bad data. Why would a reader need to run
validation on every element just in case a writer had a bug? And why should
the potential for writer bugs mean that it has to be the reader's problem?
If a writer produces bad data, we can absolutely reject it and tell the
writer to fix the bug.

I'm also puzzled by the assertion that writer guarantees have value, but
only if there are other, more specific guarantees in addition. This seems
to cede the point that a contract between readers and writers makes sense,
but only if it is a _specific_ contract. But the proposed additions don't
make sense in the same way.

Take, for example, the suggestion to also disallow zero-length vectors. At
a glance this seems similar, but I think it assumes a specific context and
is better suited to be a constraint for some use cases rather than part of
what is considered valid for the type. That's because zero-length vectors
are valid vectors with a length. You can calculate Euclidean or Manhattan
distance and use them in many contexts. They don't work if you're
normalizing to a length of 1, but it's a choice to normalize and use a dot
product to calculate cosine distance. And at this point, we're talking
about the usefulness of restrictions for the type, not a Platonic ideal.

I think the restriction that makes the most sense is that vectors have a
length and can be compared in distance calculations. 0-length vectors are
valid because they point somewhere and have a length. We shouldn't restrict
uses to only cosine distance over normalized vectors. Those are use case
dependent choices and there are other valid ones. I also don't think that
the model used to produce a vector is part of the vector type, just like
the method used to produce a metric doesn't define its type: cosine
similarity produces values between -1 and 1, but we store it in a `float`,
not a `float(cosine)`.

Why does it make sense to have a finiteness restriction as part of the
type? Because that's what a (mathematical) vector is. A simple search gives
a pretty common definition: "A vector is an object that has both magnitude
and a direction". If a dimension/element is NaN, null, or infinite, the
vector doesn't point anywhere and it doesn't have a magnitude. Not only can
it not be used, it doesn't meet the most basic definition of the type.

Ryan

On Thu, Oct 15, 2026 at 1:07 AM Jan Finis <jp...@gmail.com> wrote:

> >
> >
> >
> Daniel,
>
> alright, so no physical optimizations intended.  We only want this for
> semantic reasons, not for any significant physical optimization. If we're
> all on the same page here, that is good 👍.
>
> To phrase my remaining problem with this more concretely:
> *Let's say we add this type. What cool new stuff can we now do that we
> couldn't do before?*
> That is the litmus test that a type has to pass to be useful IMHO. And here
> I can't come up with a tangible use case that wasn't possible before.
> Especially if we one day have fixed size lists in Parquet, the remaining
> niche for this type becomes so small, that the added complexity isn't worth
> the benefit IMHO.
>
> As I guess most people on here are active maintainers of a Parquet
> implementation: Concretely, what would you be using this type for if we had
> it? Would you allow math only on fields tagged with this type? What if that
> math yields INF or NaN? Not allowed?
>
> Especially if we also discuss having both VECTOR and FINITE_VECTOR. How
> will you guys wire this up with the type systems of your engines? Do your
> engines also have two types here? Do you type-decay a FINITE_VECTOR to a
> VECTOR once someone does a multiplication that could yield INF? By itself,
> a FINITE_VECTOR sounds good. Holistically looked at from a type system
> perspective, it comes with a lot of complexity. Do the benefits really
> outweigh the complexity here? That's why I would rather have the bare
> minimum: A fixed size array (list) of floats or at least just a vector
> without finite. Then all these detailed questions go away.
>
> I'll take this one step further... Why is there any debate about this at
> > all? Why can't a type simply be added to the *specification* as an
> > administrative detail and be done?
>
>
> Andrew,
>
> Parquet is supposed to be a lingua franca of data. The specification is the
> authoritative truth for this language and all implementations are expected
> to implement as much as possible from this spec, so anything in the spec
> does causes implementation effort. So just adding anything to this spec
> without debate is not a good idea. It is even the other way around: We need
> to have way higher rigor with the spec than we need to have with any
> implementations. Implementations can be changed, the spec needs to be
> stable.
>
> A tag-only type that doesn't come with any restrictions can be added with
> little effort. But here we want to tie restrictions to it (no NaN/Inf), so
> we would expect implementations to enforce these. So there is some
> implementation effort expected here.
>
> Also, even if we only wanted a tag type, I would still argue that you
> shouldn't add those without a debate. What if A wants XML, B wants YAML, C
> wants quaternions, D wants address, E wants phone number. If we just added
> any types that are "just tags", Parquet logical types would just become a
> trash dump of any type that anyone ever thought to be useful. So rigorous
> assessment of each new type candidate is necessary.
>
> Finally, many engines couple parquet with another type system (e.g., Arrow
> or SQL). So even with a tag only type, you at least still have the
> implementation burden of wiring this up with your engine's type system.
>
> So, just because logical types are "just" logical, that doesn't mean we can
> add them without any rigor.
>
>
> A few naive questions:
> > 1. Would a physical definition with NANs & Inf+/- allow efficient
> > encodings like ALP?
> > 2. If the physical representation allows values that make encodings like
> > ALP impossible, can a tighter Logical type allow different encodings?
> > 3. Can we RLE or binpack a vector?
>
>
> The simple general answer here is that all these things do not require a
> logical type. Writers can already do any such optimization if the actual
> data contains no NaNs & Infs. All these physical optimizations just require
> no actual NaNs/Infs to be present in a page, they don't need a guarantee
> that the whole file won't contain any.
>
> Cheers,
> Jan
>
>
>
>
> Am Do., 15. Okt. 2026 um 04:28 Uhr schrieb <ku...@gmail.com>:
>
> > Hi Everyone,
> >
> > Glad to see a Vector type being added. I have a few use cases I want to
> > pursue once we have an agreed upon definition.
> >
> > In regards to that definition discussion. I feel comfortable with the
> idea
> > of the semantic meaning Dan raises though I can’t shake a slight
> discomfort
> > (maybe FOMO) with limiting values in the physical definition.
> >
> > A few naive questions:
> > 1. Would a physical definition with NANs & Inf+/- allow efficient
> > encodings like ALP?
> > 2. If the physical representation allows values that make encodings like
> > ALP impossible, can a tighter Logical type allow different encodings?
> > 3. Can we RLE or binpack a vector?
> >
> > Best Regards,
> > Kurtis C. Wright
> > > On Oct 14, 2026, at 12:45, Daniel Weeks <dw...@apache.org> wrote:
> > >
> > > Hey Jan,
> > >
> > > There's a lot that I disagree with respect to the framing of these
> > > arguments and how we should be thinking about introducing vector as a
> > > logical type.  There's a repeated tendency to let the physical
> > > representation or performance drive discussion around types as opposed
> to
> > > what it represents.
> > >
> > > In parquet, establishing semantic meaning is the purpose of logical
> > types.
> > > The logical type separates how to interpret a value from its physical
> > > representation.  We can represent a timestamp with INT64 or even a
> string
> > > value, so why have a type for it? Even though a FIXED_SIZED_LIST and
> > VECTOR
> > > may have the exact same representation and performance, they have
> > different
> > > semantic meaning.  Databases and engines across the ecosystem already
> > > recognize this differentiation and have separate representations.  The
> > > choice to separate these isn't because the discussion is too tedious;
> > > they're fundamentally different things.
> > >
> > > The finite requirement is what makes a vector a reflection of what it
> > > represents.  As we've discussed in the thread, non-finite values in the
> > > representation violate the properties of a vector and the value ceases
> to
> > > be a vector.  On the other hand, a 0 vector is still a vector (you can
> > > argue the utility of it, but it is a valid vector).  Implementations
> can
> > > choose what they validate or how to handle execution errors; that's an
> > > implementation concern, not a requirement.  Operations performed on a
> > > vector depend on the context in which they are used, but the
> > representation
> > > doesn't need to solve scenarios where that application doesn't make
> > sense.
> > >
> > > The guarantees or context needed to use a vector do not need to be
> > inherent
> > > in the type for it to be a useful type.  We don't need the currency
> used
> > > for decimal or whether a floating point is a normalized value as part
> of
> > > the type to make use of those types.  This conflates the use cases with
> > the
> > > type system and unnecessarily attempts to draw the context into the
> type
> > > definition.  Similarly, you don't need to capture normalization, model,
> > > etc. as part of the type for it to be useful.
> > >
> > > So the reason we want to add a vector type (or FINITE_VECTOR) is
> because
> > we
> > > want to store vectors with the semantic meaning and requirements based
> on
> > > what they represent.
> > >
> > > -Dan
> > >
> > >
> > >> On Wed, Oct 14, 2026 at 1:22 AM Jan Finis <jp...@gmail.com> wrote:
> > >>
> > >> A bit late to the party, but here we go:
> > >>
> > >> I have the strong sentiment that the FINITE_VECTOR type as it is
> > currently
> > >> being discussed will not be very useful and just increase
> implementation
> > >> complexity. The arguments for the type are all quite weak (details see
> > >> below). The FINITE part doesn't seem to yield any value and is not a
> > useful
> > >> restriction by itself. The vector (i.e. fixed length) part should
> > rather be
> > >> generalized instead of making a one-off thing. In general, the
> existing
> > >> types are already working well for uncompressed vectors. The real
> > challenge
> > >> that would bring real benefits is supporting quantization. This is
> where
> > >> you get orders of magnitude improvements to storage space and
> processing
> > >> speed.
> > >>
> > >> Let's first discuss the FINITE part. So far, the only arguments I have
> > read
> > >> for this are:
> > >>
> > >> 1. many other vector databases have it
> > >> 2. vectors are the hot stuff right now, so Parquet better boast having
> > >> "native" type support for vectors
> > >> 3. by focussing on this common special case, we can make rapid
> progress,
> > >> while a general solution would take too long to specify
> > >> 4. to give a guarantee to readers that no INF/NaN/NULL will be in the
> > >> vector
> > >>
> > >> the first two arguments are quite weak by themselves. Vector databases
> > are
> > >> very specialized engines. Such engines can of course be very
> > opinionated on
> > >> their data types. A general format like Parquet must instead weigh
> > whether
> > >> a special type is worth the added complexity over a more general,
> widely
> > >> applicable type. In this case, it is IMHO not at all. And just having
> a
> > >> type called "vector" for advertisement reasons to make Parquet appear
> > more
> > >> modern seems a non-goal in the first place.
> > >>
> > >> The third "time to market" argument is valid for systems that don't
> > need a
> > >> stable interface. It is a very dangerous attitude for formats like
> > Parquet
> > >> that must be stable long term and should have a consolidated, concise,
> > >> non-redundant type system. For such long term storage formats, we need
> > to
> > >> take the time to build the right thing, as any shortcuts we take will
> > stay
> > >> as blemishes in the format forever. If I read the discussion
> correctly,
> > >> this is actually the main argument as of now: We have found that it is
> > hard
> > >> to align on how a fixed size list type should look like, so we want to
> > go
> > >> the easier route of a specialized type. This seems like a
> > >> dangerous shortcut fueled by feature recency bias. Ask yourself: Would
> > we
> > >> want this, if Parquet already had a fixed size list type?
> > >>
> > >> So finally, the only technical argument is the fourth one: "It signals
> > >> semantics and ensures that no bad data lands in the format". I see two
> > big
> > >> problems with this:
> > >>
> > >> 1. Readers *still* need to test for NaN / INF because Parquet files
> may
> > be
> > >> corrupt. Nothing can stop a malicious or buggy writer from physically
> > >> writing NaN or INF. *So you save no single line of code in a reader.
> You
> > >> save no branches, as you still need to handle the INF/NaN case.* So
> this
> > >> doesn't help with performance at all. The only difference is, that the
> > >> error message changes from "found an invalid INF value in the data" to
> > >> "Your parquet file seems to be corrupt". It is actually quite the
> > opposite,
> > >> readers and writers get more complex having to handle this new type.
> > >>
> > >> If you now argue that it's not about performance, it's about defined
> > >> semantics and erroring out early in writers, then the second problem
> > >> arises:
> > >>
> > >> 2. The proposed no-NaN/INF/NULL guarantee is just one of multiple
> > >> guarantees that you need for sensible vector processing. If "no
> > >> INF/NaN/NULL" was the only guarantee you needed, I would agree that
> > >> FINITE_VECTOR is useful, but it is by no means the only guarantee you
> > need,
> > >> instead, depending on the distance metric, you also need the following
> > >> guarantees. E.g.:
> > >>
> > >>   - No vector may be the 0 vector, otherwise, e.g., cosine similarity
> is
> > >>   undefined. Cosine similarity is often used as default, so this is
> > >> actually
> > >>   quite important in practice.
> > >>   - If we want to use a dot product, then all vectors need to
> guarantee
> > a
> > >>   length of 1. This would actually be quite useful for performance
> > >> reasons.
> > >>   - The vectors must come from the same model as the vector used for
> the
> > >>   search. Otherwise, the search yields a numeric result, but that
> result
> > >> has
> > >>   no sensible meaning.
> > >>
> > >> So, what use is a type that just guarantees part of the things you
> need
> > for
> > >> vector search but not others? You still need additional guarantees in
> > >> addition to that type. What makes INF/NaN so important in comparison
> to,
> > >> say, "no 0 vector present", that it warrants an own type, while the
> > other
> > >> guarantees do not? If we really want this type as a basis for
> > cross-engine
> > >> sharing of vectors, then it needs to give *all* guarantees.
> > >>
> > >> I'm not against a specialized vector type if it really carries *all*
> > >> information to be useful by itself. If it carried all that
> information,
> > >> then a reading engine could indeed validate that only meaningful
> vector
> > >> searches are executed. But as long as it is just a half-baked thing,
> it
> > is
> > >> not useful by itself and just a burden on each implementer and
> > maintainer.
> > >> If we need extra information anyway, why not just stick to a general
> > type
> > >> like VECTOR (without FINITE).
> > >>
> > >> If you're a proponent of FINITE_VECTOR, ask yourself (or tell us) what
> > >> advantage you will *really* get over just a (potentially fixed size)
> > List
> > >> of Float? As discussed, it's not performance. It's also not a semantic
> > >> marker that is sufficient by itself. It's just a small hint that
> allows
> > >> writers to error out once they see a NaN or INF in the input, while
> they
> > >> will still happily write a bogus 0 vector. This seems to be of zero
> > >> benefit.
> > >>
> > >> Anecdotally, we currently handle petabytes of vectors in production in
> > >> Data360. We simply store them in a List of Float, with an optimized
> > R-Level
> > >> reader implementation that rapidly checks that all vectors are the
> same
> > >> length. This solution works just fine. The overhead of R-level
> decoding
> > is
> > >> barely measurable. The check is super fast and R-levels are tiny in
> > >> comparison to the payload vectors, so also the added I/O is
> negligible.
> > >> This is also why I'm so firm in my sentiment: We are running a
> > production
> > >> grade system processing trillions of vectors right now and see no big
> > >> issues with the types that Parquet currently provides.
> > >>
> > >> As I don't just want to be a naysayer and point out problems with the
> > >> current design, here are the directions that I would propose indeed:
> > >>
> > >>   - a general fixed size list type. Don't do shortcuts just because
> the
> > >>   discussion is too tedious.
> > >>   - a vector type that indeed carries enough information to be really
> > >>   useful for vector search by itself. (e.g., model name / version,
> > >> optional
> > >>   guarantees for no 0 vector / normalized vector length). With such a
> > >> type,
> > >>   you could really do well defined vector searches even if the writing
> > >> engine
> > >>   is completely different to the reading one.
> > >>   - Standardize quantizations. This will be a real game changer for
> > >>   performance.
> > >>
> > >> Cheers,
> > >> Jan
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>> Am Mi., 14. Okt. 2026 um 03:36 Uhr schrieb Gang Wu <ustcwg@gmail.com
> >:
> > >>>
> > >>> I think separating them into FINITE_VECTOR and FIXED_LENGTH_LIST is a
> > >> good
> > >>> idea, though I think renaming FINITE_VECTOR to VECTOR looks more
> > natural.
> > >>> Should we move forward with proposing FINITE_VECTOR/VECTOR first?
> > >>>
> > >>> On Tue, Oct 13, 2026 at 6:26 PM Alkis Evlogimenos via dev <
> > >>> dev@parquet.apache.org> wrote:
> > >>>
> > >>>> I suggest we go with VECTOR for the finite vector to keep the name
> the
> > >>> same
> > >>>> as other engines/type systems. If we want a non-finite one in the
> > >> future
> > >>> we
> > >>>> can name that something else or we add fixed size list that will
> serve
> > >>> the
> > >>>> same purpose without a special name.
> > >>>>
> > >>>>
> > >>>> On Thu, Oct 8, 2026 at 10:20 PM Ryan Blue <bl...@apache.org> wrote:
> > >>>>
> > >>>>> A logical type would:
> > >>>>> - Describe data for users and readers to understand what the file
> > >>>> contains
> > >>>>> - Signal that repetition and definition levels will reflect a fixed
> > >>>> number
> > >>>>> of elements, which implementations can use to optimize decoding
> > >>>>> - Guarantee that there are no null elements (because the type
> cannot
> > >> be
> > >>>>> applied to a list with optional elements)
> > >>>>> - [FINITE_VECTOR] Guarantee that no elements are NaN or Infinity
> > >> values
> > >>>> so
> > >>>>> that the vectors have a finite length and are suitable for distance
> > >>>>> calculations
> > >>>>>
> > >>>>> The finite value restriction is a restriction on writers to prevent
> > >> bad
> > >>>>> data from being written.
> > >>>>>
> > >>>>> Ignoring logical types that are not understood is a rule for
> > >> readers. A
> > >>>>> reader can handle vectors and pass it on to another layer. For
> > >>> instance,
> > >>>> a
> > >>>>> Parquet reader that doesn't support geometry passes values as
> > >> `binary`.
> > >>>>> That binary value can be passed to a geometry UDF, even if the
> > >>> underlying
> > >>>>> engine and Parquet reader doesn't have support.
> > >>>>>
> > >>>>> A tool that doesn't understand FINITE_VECTOR can similarly pass a
> > >> list
> > >>> of
> > >>>>> floats along to a UDF. But it is still important to the user that
> bad
> > >>>>> values are not present in the data. In your example, bad values are
> > >>>> vectors
> > >>>>> that cause the centroid calculation to fail, like vectors with
> > >> infinite
> > >>>>> length.
> > >>>>>
> > >>>>> Ryan
> > >>>>>
> > >>>>> On Thu, Oct 8, 2026 at 12:54 PM Will Edwards via dev <
> > >>>>> dev@parquet.apache.org>
> > >>>>> wrote:
> > >>>>>
> > >>>>>> Could we clarify the use cases for a VECTOR annotation on
> > >>> LIST<FLOAT>?
> > >>>>>> Specifically, what does it make possible that's impossible today,
> > >> and
> > >>>>> what
> > >>>>>> does it make easier that's currently hard?
> > >>>>>>
> > >>>>>> I'm struggling to find a strong answer. I'd like every engine to
> > >>> offer
> > >>>>>> vector functions on lists of floats without needing a new logical
> > >>> type.
> > >>>>>>
> > >>>>>> To be clear, I'm still enthusiastic about a vector type that
> > >>> describes
> > >>>>> how
> > >>>>>> to unpack and scale composite representations (quantised codes
> plus
> > >>>>> scales,
> > >>>>>> codebooks, etc.), in the direction ML is going. That adds
> > >> information
> > >>>>>> engines don't have. A label on LIST<FLOAT> doesn't: engines
> already
> > >>>>>> understand the physical type, and many already offer vector
> > >> functions
> > >>>> on
> > >>>>>> it. I don't see a performance benefit, and NaN/null/Inf checks are
> > >>>> cheap
> > >>>>>> enough that engines should do them anyway. The label also doesn't
> > >> say
> > >>>>> what
> > >>>>>> you can do with the column. Knowing that a column holds embeddings
> > >>>>> doesn't
> > >>>>>> say how they can be used, just as zipcode and age are both
> integers
> > >>> but
> > >>>>>> multiplying them is meaningless.
> > >>>>>>
> > >>>>>> My concern is the opposite effect. We're establishing that engines
> > >>>> should
> > >>>>>> ignore logical types they don't understand. So a tool that doesn't
> > >>> know
> > >>>>>> VECTOR will read the column as LIST<FLOAT>, let the user compute
> > >>>>> centroids
> > >>>>>> (say), and write plain LIST<FLOAT> back out. If other engines
> > >>> restrict
> > >>>>>> embedding functions to VECTOR-annotated columns, that file stops
> > >>>> working
> > >>>>>> with them, through no fault of the user. An annotation that
> doesn't
> > >>>>> change
> > >>>>>> what the data is shouldn't be able to make it unusable.
> > >>>>>>
> > >>>>>> Am I missing a use case?
> > >>>>>>
> > >>>>>> Best, Will
> > >>>>>>
> > >>>>>> On Thu, 8 Oct 2026 at 19:53, Steven Wu <st...@gmail.com>
> > >> wrote:
> > >>>>>>
> > >>>>>>> I also think a FINITE_VECTOR type brings a lot of values to many
> > >>> use
> > >>>>>> cases.
> > >>>>>>> It is would be great to make progress on the FINITE_VECTOR, and
> > >> not
> > >>>> to
> > >>>>>> fit
> > >>>>>>> every use cases into the FIXED_LENGTH_LIST.
> > >>>>>>>
> > >>>>>>> On Mon, Oct 5, 2026 at 1:26 PM Ryan Blue <bl...@apache.org>
> > >> wrote:
> > >>>>>>>
> > >>>>>>>> Going through this thread again, it looks like there are a few
> > >>>> things
> > >>>>>>> that
> > >>>>>>>> people generally agree on:
> > >>>>>>>>
> > >>>>>>>> - In the short term, we want to introduce a logical type or
> > >>> types;
> > >>>> a
> > >>>>>>> better
> > >>>>>>>> physical representation can be done later. We can focus on
> > >>> logical
> > >>>>>> types.
> > >>>>>>>> - There is precedent in logical types for restricting values,
> > >>> like
> > >>>>> UTF8
> > >>>>>>>> strings to contain only UTF-8 codepoints. I think we also agree
> > >>>> that
> > >>>>> we
> > >>>>>>>> could restrict vector elements to non-nested or primitive
> > >> types.
> > >>>>>>>> - There are many examples of databases that restrict their
> > >> vector
> > >>>>> types
> > >>>>>>> to
> > >>>>>>>> non-null, finite elements.
> > >>>>>>>> - Some people want to have write-time guarantees that vectors
> > >>> have
> > >>>>> only
> > >>>>>>>> non-null, finite elements
> > >>>>>>>> - Some people want to extend this work to include tensors,
> > >>> shorter
> > >>>>>>> floating
> > >>>>>>>> point representations, variable lengths, NaN or Infinity
> > >>> elements,
> > >>>>> etc.
> > >>>>>>>>
> > >>>>>>>> It sounds like we are nearly there, but have two halves of the
> > >>>>>> community
> > >>>>>>>> that care about different things. That just means we should
> > >> have
> > >>>>>> multiple
> > >>>>>>>> types. It's entirely reasonable to want a type that allows NaN
> > >>>>> values,
> > >>>>>>> and
> > >>>>>>>> it's also reasonable for those of us building databases, table
> > >>>>>> standards,
> > >>>>>>>> and vector indexes to care about guarantees that elements are
> > >>>> finite.
> > >>>>>>>> There's more than enough examples to show that this is a
> > >>> legitimate
> > >>>>> use
> > >>>>>>>> case.
> > >>>>>>>>
> > >>>>>>>> Since I don't see the concerns of the different use cases
> > >>> changing,
> > >>>>> it
> > >>>>>>>> looks like we need to find a compromise to move forward. I
> > >>> propose
> > >>>>> that
> > >>>>>>> we
> > >>>>>>>> introduce a FINITE_VECTOR type and separately a
> > >> FIXED_LENGTH_LIST
> > >>>> or
> > >>>>>>>> another VECTOR type that allows infinite and NaN values.
> > >>>>>>>>
> > >>>>>>>> Can we agree on a compromise? Is there a strong reason to try
> > >> to
> > >>>> fit
> > >>>>>>> these
> > >>>>>>>> use cases into a single type? What if that means not making
> > >>>> progress?
> > >>>>>>>>
> > >>>>>>>> Ryan
> > >>>>>>>>
> > >>>>>>>> On Mon, Oct 5, 2026 at 12:15 PM Philipp Fischbeck via dev <
> > >>>>>>>> dev@parquet.apache.org> wrote:
> > >>>>>>>>
> > >>>>>>>>> Thinking about this more, I am also fine with pulling out the
> > >>>>>> “NaN/Inf”
> > >>>>>>>>> requirement. Either into a type boolean (my preference), or
> > >>>> putting
> > >>>>>> no
> > >>>>>>>>> restrictions on the Parquet level, allowing all IEEE 754
> > >> float
> > >>>>>> numbers.
> > >>>>>>>> If
> > >>>>>>>>> some top-level application wants to put additional
> > >> constraints
> > >>> on
> > >>>>> the
> > >>>>>>>> data
> > >>>>>>>>> type, like Iceberg not allowing NaN/infinity, it can still do
> > >>> so
> > >>>>> one
> > >>>>>>>> layer
> > >>>>>>>>> above. It seems like this was the biggest point of discussion
> > >>>>>>> remaining.
> > >>>>>>>>> All these options work for me, and we should move forward
> > >> with
> > >>>> this
> > >>>>>>>>> proposal.
> > >>>>>>>>>
> > >>>>>>>>> Concerning additional element types like bf16, they can be
> > >>>>> introduced
> > >>>>>>> as
> > >>>>>>>>> Parquet types (at least in the current LIST-based spec
> > >>> proposal).
> > >>>>>>>>>
> > >>>>>>>>> Concerning Will’s suggestion around representing VECTOR as a
> > >>>>> variable
> > >>>>>>>>> length list with a MIN/MAX_LENGTH metadata restriction: I
> > >> think
> > >>>>> that
> > >>>>>> a
> > >>>>>>>>> dedicated logical VECTOR type still provides enough benefits
> > >> by
> > >>>>>>>> describing
> > >>>>>>>>> the semantics of the data, and allowing to capture future
> > >>> changes
> > >>>>>>> around
> > >>>>>>>>> statistics. In that case, the length is better captured as an
> > >>>> exact
> > >>>>>>>> length
> > >>>>>>>>> for this type specifically. I’m also not sure about other use
> > >>>> cases
> > >>>>>> for
> > >>>>>>>>> MIN_LENGTH, so introducing a new type constraint system just
> > >>> for
> > >>>>> the
> > >>>>>>>>> MIN_LENGTH=MAX_LENGTH case for vectors doesn’t have enough
> > >>> value
> > >>>> in
> > >>>>>> my
> > >>>>>>>>> view.
> > >>>>>>>>>
> > >>>>>>>>> Best,
> > >>>>>>>>> Philipp
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>> On Thu, Oct 1, 2026 at 9:00 AM Will Edwards via dev <
> > >>>>>>>>> dev@parquet.apache.org>
> > >>>>>>>>> wrote:
> > >>>>>>>>>
> > >>>>>>>>>> +1 Sebastian, a vector type suggests usage beyond
> > >> normalized
> > >>>>>>> embeddings
> > >>>>>>>>> and
> > >>>>>>>>>> engines really aren't going to be able to elide all the
> > >>> checks
> > >>>>>>> anyway.
> > >>>>>>>>>>
> > >>>>>>>>>> Antoine, you suggested we explore metadata solutions too :)
> > >>>>>>>>>>
> > >>>>>>>>>> Today, a database supporting CHAR(N) or VARCHAR(n) can't
> > >>>> express
> > >>>>>> this
> > >>>>>>>> in
> > >>>>>>>>>> Parquet.  Engines can write opaque blobs they understand in
> > >>> the
> > >>>>>>>> metadata
> > >>>>>>>>>> (e.g. Arrow does) so the extra type info round-trips, but
> > >>> this
> > >>>>>> isn't
> > >>>>>>>>>> standardized, portable, nor interoperable.
> > >>>>>>>>>>
> > >>>>>>>>>> So perhaps standardizing constraint metadata or new Thrift
> > >>>> fields
> > >>>>>> (my
> > >>>>>>>>> pref)
> > >>>>>>>>>> can generically support VECTOR(N) whilst also being more
> > >>>> broadly
> > >>>>>>> useful
> > >>>>>>>>> for
> > >>>>>>>>>> other use cases and data types?
> > >>>>>>>>>>
> > >>>>>>>>>> Imagine metadata or Thrift fields to tag the MIN_LENGTH and
> > >>>>>>> MAX_LENGTH
> > >>>>>>>>> for
> > >>>>>>>>>> Parquet's variable-length physcial types.  These would
> > >> count
> > >>>>>> logical
> > >>>>>>>>> units
> > >>>>>>>>>> (e.g. UTF8 code points for STRING) rather than bytes.
> > >>>>>>>>>>
> > >>>>>>>>>> Then one engine's VECTOR(N) can be written to Parquet as
> > >>>> LIST(N)
> > >>>>>> with
> > >>>>>>>>>> MIN_LENGTH = MAX_LENGTH = N and then readers - even
> > >> different
> > >>>>>>> engines -
> > >>>>>>>>> can
> > >>>>>>>>>> pick up these constraints if they support doing so.  We
> > >> don't
> > >>>>>>> actually
> > >>>>>>>>> need
> > >>>>>>>>>> a logical type to express a VECTOR then.
> > >>>>>>>>>>
> > >>>>>>>>>> This also allows standard SQL's CHAR(N) and VARCHAR(N) to
> > >> be
> > >>>>>>> expressed
> > >>>>>>>> on
> > >>>>>>>>>> UTF8 columns etc.
> > >>>>>>>>>>
> > >>>>>>>>>> This also seems generally useful in the future as we
> > >> approach
> > >>>>>>>> expressing
> > >>>>>>>>>> other tensor data shapes and types.
> > >>>>>>>>>>
> > >>>>>>>>>> This won't change the mechanics: the storage type doesn't
> > >>>> enforce
> > >>>>>>> these
> > >>>>>>>>>> constraints and engines must check them. It's previously
> > >>>>>> established
> > >>>>>>>> that
> > >>>>>>>>>> performance is just a software problem and there is no
> > >>>>> fundamental
> > >>>>>>>>>> performance advantage to the fixed size types in
> > >> Dremel-like
> > >>>>>> formats
> > >>>>>>>> like
> > >>>>>>>>>> Parquet.
> > >>>>>>>>>>
> > >>>>>>>>>> Best,
> > >>>>>>>>>> Will
> > >>>>>>>>>>
> > >>>>>>>>>> On Wed, 30 Sept 2026 at 16:08, Sebastian Baunsgaard <
> > >>>>>>>>> baunsgaard@apache.org
> > >>>>>>>>>>>
> > >>>>>>>>>> wrote:
> > >>>>>>>>>>
> > >>>>>>>>>>> After listening in on the last meeting Wednesday, I have
> > >>> been
> > >>>>>>>> thinking
> > >>>>>>>>>>> about this quite a bit, and thought I would share my 2
> > >>> cents.
> > >>>>>>>>>>>
> > >>>>>>>>>>> I suggest not requiring VECTOR elements to be finite, as
> > >>>>> proposed
> > >>>>>>> in
> > >>>>>>>>> the
> > >>>>>>>>>> PR
> > >>>>>>>>>>> <https://github.com/apache/parquet-format/pull/624>.
> > >>>>>>>>>>>
> > >>>>>>>>>>> While Divjot and others note the specialization for
> > >> UTF-8,
> > >>>>>> Parquet
> > >>>>>>>>>> already
> > >>>>>>>>>>> defines DOUBLE and FLOAT as IEEE encodings
> > >>>>>>>>>>> <
> > >>>>>>>>
> > >>>>
> https://github.com/apache/parquet-format/blob/master/README.md#types
> > >>>>>>>>>> .
> > >>>>>>>>>>>
> > >>>>>>>>>>> Making NaN or +/-Inf illegal inside VECTOR would be a new
> > >>>>>>> restriction
> > >>>>>>>>> on
> > >>>>>>>>>>> those same underlying physical types. Scalar FLOAT/DOUBLE
> > >>>>> columns
> > >>>>>>>> have
> > >>>>>>>>>>> never had such a rule.
> > >>>>>>>>>>>
> > >>>>>>>>>>> It would also not match how the systems that actually
> > >>> produce
> > >>>>>> these
> > >>>>>>>>>> vectors
> > >>>>>>>>>>> treat floats. Most of them use IEEE floats, e.g. Spark,
> > >>>> DuckDB,
> > >>>>>>>> NumPy,
> > >>>>>>>>>>> PyTorch, TensorFlow, or lower level CUDA, oneMKL, and
> > >>>> OpenBLAS.
> > >>>>>>>>>>>
> > >>>>>>>>>>> pgvector's "all elements must be finite" rule is an
> > >> insert
> > >>>>>> policy.
> > >>>>>>>>> While
> > >>>>>>>>>> I
> > >>>>>>>>>>> think Parquet as a general-purpose format should allow
> > >> them
> > >>>> out
> > >>>>>> of
> > >>>>>>>> the
> > >>>>>>>>>> box.
> > >>>>>>>>>>> It is a user system decision to disallow writing these
> > >>> values
> > >>>>> or
> > >>>>>> to
> > >>>>>>>>> throw
> > >>>>>>>>>>> errors when encountering them.
> > >>>>>>>>>>>
> > >>>>>>>>>>> A type-level ban also does not keep Inf/NaN out of vector
> > >>>> math.
> > >>>>>>>> Finite
> > >>>>>>>>>>> coordinates still overflow. For instance: magnitude =
> > >>>>>> ||[FLOAT_MAX,
> > >>>>>>>>>>> FLOAT_MAX]|| where every coordinate is finite overflows
> > >> to
> > >>>>> +Inf.
> > >>>>>>>>> PyTorch
> > >>>>>>>>>>> documents the same overflow: tensor([1e20, 1e20]).norm()
> > >> is
> > >>>> inf
> > >>>>>> in
> > >>>>>>>>> fp32.
> > >>>>>>>>>>>
> > >>>>>>>>>>> A generated MAGNITUDE(vec) column would then be Inf while
> > >>> the
> > >>>>>>> VECTOR
> > >>>>>>>>>> itself
> > >>>>>>>>>>> was required to be finite. The ban only forbids storing
> > >>>>> specials
> > >>>>>> in
> > >>>>>>>> the
> > >>>>>>>>>>> coordinates, and does not prevent even simple downstream
> > >>>>>>> computations
> > >>>>>>>>>> from
> > >>>>>>>>>>> producing them.
> > >>>>>>>>>>>
> > >>>>>>>>>>> I suggest keeping VECTOR as the type and add an optional
> > >>> bool
> > >>>>>>>>>>> VectorType.finite (default false). Engines that want
> > >>>>>> pgvector-like
> > >>>>>>>>>>> rejection can require the flag = true. Engines that use
> > >>>> NumPy /
> > >>>>>>>>> PyTorch /
> > >>>>>>>>>>> Spark arrays can keep representable IEEE values. For the
> > >>> same
> > >>>>>>>> reason, I
> > >>>>>>>>>> do
> > >>>>>>>>>>> not favor adding yet another type called FINITE_VECTOR.
> > >>>>>>>>>>>
> > >>>>>>>>>>> Best,
> > >>>>>>>>>>> Sebastian
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>> On Tue, Sep 29, 2026 at 2:03 PM Divjot Arora via dev <
> > >>>>>>>>>>> dev@parquet.apache.org>
> > >>>>>>>>>>> wrote:
> > >>>>>>>>>>>
> > >>>>>>>>>>>> I agree with moving forward with the proposal and
> > >>> renaming
> > >>>>>> VECTOR
> > >>>>>>>> to
> > >>>>>>>>>>>> FINITE_VECTOR or something
> > >>>>>>>>>>>> similar if others feel that better communicates its
> > >>>> intent. I
> > >>>>>> see
> > >>>>>>>>> value
> > >>>>>>>>>>> in
> > >>>>>>>>>>>> having a vector type that guarantees
> > >>>>>>>>>>>> its values are mathematically-correct vectors for a few
> > >>>>>> reasons:
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> 1. As others have mentioned, this is similar to the
> > >>> STRING
> > >>>>>>> logical
> > >>>>>>>>> type
> > >>>>>>>>>>>> mandating UTF-8.
> > >>>>>>>>>>>> 2. It is not possible to rely on statistics for
> > >> detecting
> > >>>> NaN
> > >>>>>> or
> > >>>>>>>>> +-inf
> > >>>>>>>>>>>> values as all statistics are optional.
> > >>>>>>>>>>>> To my knowledge, there is no precedent for requiring
> > >> that
> > >>>>>> writers
> > >>>>>>>>> emit
> > >>>>>>>>>>>> stats for specific logical types.
> > >>>>>>>>>>>> 3. Without statistics, NaN/inf values could flow
> > >> through
> > >>>> the
> > >>>>>>> engine
> > >>>>>>>>> and
> > >>>>>>>>>>>> block them for specific functionality.
> > >>>>>>>>>>>> This seems to be Lance's behavior. However, I don't
> > >> think
> > >>>>> this
> > >>>>>> is
> > >>>>>>>> as
> > >>>>>>>>>> easy
> > >>>>>>>>>>>> as it sounds and Lance has run
> > >>>>>>>>>>>> into several bugs due to this ([1] [2] for examples).
> > >>> Their
> > >>>>>>> Python
> > >>>>>>>>> [3]
> > >>>>>>>>>>> and
> > >>>>>>>>>>>> Rust [4] APIs default to erroring on
> > >>>>>>>>>>>> vectors that contain NaN values.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> I agree that Parquet is a general-purpose format, but I
> > >>>> don't
> > >>>>>>> think
> > >>>>>>>>>> this
> > >>>>>>>>>>>> implies the lack of domain-specific types.
> > >>>>>>>>>>>> Instead, this broadens the use cases where Parquet
> > >> would
> > >>> be
> > >>>>>>>>> considered
> > >>>>>>>>>> a
> > >>>>>>>>>>>> good fit. I also fully agree with
> > >>>>>>>>>>>> continuing work on a general fixed-size list type if we
> > >>>>> believe
> > >>>>>>>> there
> > >>>>>>>>>> are
> > >>>>>>>>>>>> non-vector use cases that would benefit
> > >>>>>>>>>>>> from such guarantees at the storage layer.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> [1] https://github.com/lance-format/lance/pull/3648
> > >>>>>>>>>>>> [2] https://github.com/lance-format/lance/issues/8607
> > >>> (see
> > >>>>>>> point 2
> > >>>>>>>>> in
> > >>>>>>>>>>>> "root
> > >>>>>>>>>>>> cause")
> > >>>>>>>>>>>> [3]
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>>
> > >>>>
> > >>>
> > >>
> >
> https://github.com/lancedb/lancedb/blob/0327618a62483feb6dca2336549845e5887cd48f/python/python/lancedb/table.py#L4807-L4911
> > >>>>>>>>>>>> [4]
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>>
> > >>>>
> > >>>
> > >>
> >
> https://github.com/lancedb/lancedb/blob/0327618a62483feb6dca2336549845e5887cd48f/rust/lancedb/src/table/add_data.rs#L105-L110
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> On Mon, Sep 28, 2026 at 6:19 PM Philipp Fischbeck via
> > >>> dev <
> > >>>>>>>>>>>> dev@parquet.apache.org> wrote:
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>> I also favor moving forward with the proposal as is.
> > >> I
> > >>>> see
> > >>>>>>> strong
> > >>>>>>>>>> value
> > >>>>>>>>>>>> and
> > >>>>>>>>>>>>> use cases for the numeric vector type, and I agree
> > >> with
> > >>>> the
> > >>>>>>>>>>> restrictions
> > >>>>>>>>>>>>> around NaN/Inf values and element types in general.
> > >>>>> Precedent
> > >>>>>>>>> exists
> > >>>>>>>>>> in
> > >>>>>>>>>>>>> other types that restrict which physical type values
> > >>> are
> > >>>>>>>> logically
> > >>>>>>>>>>> valid
> > >>>>>>>>>>>>> (e.g., the geospatial types needing to be
> > >> WKB-encoded).
> > >>>> A
> > >>>>>>>> general
> > >>>>>>>>>>>>> fixed-size list also has value, but I think it makes
> > >>>> sense
> > >>>>> to
> > >>>>>>>>>> separate
> > >>>>>>>>>>>> the
> > >>>>>>>>>>>>> two types. The semantics are relevant, and I can see
> > >>> how
> > >>>>> the
> > >>>>>>> two
> > >>>>>>>>>> types
> > >>>>>>>>>>>>> might diverge further in the future regarding
> > >>> statistics
> > >>>> or
> > >>>>>>>>>> encodings.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> I would be okay with renaming the vector type, but it
> > >>>> seems
> > >>>>>>> like
> > >>>>>>>>> the
> > >>>>>>>>>>>> VECTOR
> > >>>>>>>>>>>>> name is already used across other engines mentioned
> > >> in
> > >>>> this
> > >>>>>>>> thread.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Best,
> > >>>>>>>>>>>>> Philipp
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> On Sat, Sep 26, 2026 at 4:37 PM Gang Wu <
> > >>>> ustcwg@gmail.com>
> > >>>>>>>> wrote:
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>> To separate orthogonal concerns, I feel we should
> > >>> treat
> > >>>>> the
> > >>>>>>>>> general
> > >>>>>>>>>>>>>> FixedSizeList as the underlying physical layout,
> > >> and
> > >>>>>> separate
> > >>>>>>>> it
> > >>>>>>>>>> from
> > >>>>>>>>>>>> the
> > >>>>>>>>>>>>>> discussion of a specialized VECTOR (or broader ML
> > >>>> tensor)
> > >>>>>>>> logical
> > >>>>>>>>>>> type.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Currently, the immediate need is to define a
> > >>>> specialized
> > >>>>>>> VECTOR
> > >>>>>>>>>> type
> > >>>>>>>>>>> so
> > >>>>>>>>>>>>>> that different engines and formats have a common
> > >>>> semantic
> > >>>>>>>>> contract
> > >>>>>>>>>>> for
> > >>>>>>>>>>>>>> interoperability.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> If the main point of contention is just the
> > >>>> NaN/Inf/null
> > >>>>>>>>>> limitations,
> > >>>>>>>>>>>>> maybe
> > >>>>>>>>>>>>>> we can resolve this by adding a qualification to
> > >> the
> > >>>> name
> > >>>>>>> (like
> > >>>>>>>>>>> Andrew
> > >>>>>>>>>>>>>> suggested, e.g., VECTOR_EMBEDDING or FINITE_VECTOR)
> > >>> to
> > >>>>>> avoid
> > >>>>>>>>>>> confusion
> > >>>>>>>>>>>>> with
> > >>>>>>>>>>>>>> broader ML use cases. Alternatively, we could
> > >>>> introduce a
> > >>>>>>>> boolean
> > >>>>>>>>>>> flag
> > >>>>>>>>>>>>>> (e.g., optional bool VectorType.finite) as proposed
> > >>>>>> earlier.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> On Fri, Sep 25, 2026 at 3:52 AM Andrew Lamb <
> > >>>>>>>>>> andrewlamb11@gmail.com>
> > >>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> I am a little confused about the points of
> > >>>> disagreement
> > >>>>>>> here.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Is it that the name `VECTOR` is too general for
> > >> the
> > >>>>>>> specific
> > >>>>>>>>> use
> > >>>>>>>>>>> case
> > >>>>>>>>>>>>> of
> > >>>>>>>>>>>>>>> fixed length arrays with non-finite values used
> > >> for
> > >>>>>> vector
> > >>>>>>>>>>>> embeddings?
> > >>>>>>>>>>>>>> If
> > >>>>>>>>>>>>>>> so, maybe we could add a qualification to the
> > >> name
> > >>>>>>>>>> (FINITE_VECTOR?
> > >>>>>>>>>>>>>>> FEATURE_VECTOR? VECTOR_EMBEDDING?)
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> It seems like there is precedent for defining
> > >>> logical
> > >>>>>> types
> > >>>>>>>>> that
> > >>>>>>>>>>>>> specify
> > >>>>>>>>>>>>>>> certain physical types and restrict their
> > >> contents
> > >>> to
> > >>>>>> help
> > >>>>>>>>>> readers
> > >>>>>>>>>>>>>>> interpret meaning, for example string[1] or JSON;
> > >>>>>> Defining
> > >>>>>>>> one
> > >>>>>>>>>> for
> > >>>>>>>>>>>>>> vector
> > >>>>>>>>>>>>>>> embeddings seems reasonable enough to me.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Andrew
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> [1]:
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>>
> > >>>>
> > >>>
> > >>
> >
> https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#string
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> On Thu, Sep 24, 2026 at 3:34 PM Daniel Weeks <
> > >>>>>>>>> dweeks@apache.org>
> > >>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> Well, Parquet is not a single-purpose format.
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> Vector stores can choose whatever policy fits
> > >>>> their
> > >>>>>>>> usage,
> > >>>>>>>>>> but
> > >>>>>>>>>>>>> we're
> > >>>>>>>>>>>>>>>>> talking about a general-purpose file format
> > >>> that
> > >>>>> aims
> > >>>>>>> to
> > >>>>>>>> be
> > >>>>>>>>>>>> broadly
> > >>>>>>>>>>>>>>>>> applicable.
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> I agree that the format is intended to be
> > >> general
> > >>>>>>> purpose,
> > >>>>>>>>> but
> > >>>>>>>>>>>> types
> > >>>>>>>>>>>>>>> within
> > >>>>>>>>>>>>>>>> the format define specific representation and
> > >>>>> semantics
> > >>>>>>> to
> > >>>>>>>>>>> achieve
> > >>>>>>>>>>>>> that
> > >>>>>>>>>>>>>>>> goal.
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> "Vector" has a more general sense than ML and
> > >>>> it's
> > >>>>>>> quite
> > >>>>>>>>>>> trivial
> > >>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>> produce vectors with non-finite values using
> > >>> e.g.
> > >>>>>> NumPy
> > >>>>>>>> or
> > >>>>>>>>>>> Pandas
> > >>>>>>>>>>>>> (or
> > >>>>>>>>>>>>>>>>> any other piece of dataframe or scientific
> > >>>>> computing
> > >>>>>>>>>> software).
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> The justification for restricting non-finite
> > >>>> values
> > >>>>> is
> > >>>>>>>>>> directly
> > >>>>>>>>>>>>>> related
> > >>>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>> the semantic meaning of a vector.  Vectors have
> > >>>>>> specific
> > >>>>>>>>>>>>>> characteristics
> > >>>>>>>>>>>>>>>> (dimension, magnitude, direction, etc).
> > >>> Non-finite
> > >>>>>>> values
> > >>>>>>>>> are
> > >>>>>>>>>>>>> invalid
> > >>>>>>>>>>>>>>> and
> > >>>>>>>>>>>>>>>> break those characteristics.  I'd liken it to
> > >>>> storing
> > >>>>>>>>> non-utf8
> > >>>>>>>>>>>> values
> > >>>>>>>>>>>>>> in
> > >>>>>>>>>>>>>>> a
> > >>>>>>>>>>>>>>>> string and still calling it a string.
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> That also differentiates a vector from a fixed
> > >>>> length
> > >>>>>>> list,
> > >>>>>>>>>> which
> > >>>>>>>>>>>> is
> > >>>>>>>>>>>>>> more
> > >>>>>>>>>>>>>>>> general and does not carry the same semantic
> > >>>> meaning.
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> I also don't think we need to bundle all
> > >>> similarly
> > >>>>>> shaped
> > >>>>>>>> use
> > >>>>>>>>>>> cases
> > >>>>>>>>>>>>>> into
> > >>>>>>>>>>>>>>>> the same solution (e.g. flxed size list).
> > >> Vector
> > >>>> is
> > >>>>> a
> > >>>>>>> well
> > >>>>>>>>>>>>> established
> > >>>>>>>>>>>>>>>> data type at this point and aligns with the
> > >>> general
> > >>>>>>> purpose
> > >>>>>>>>>>> nature
> > >>>>>>>>>>>> of
> > >>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>> format (many other formats and databases
> > >> natively
> > >>>>>> support
> > >>>>>>>>>> this).
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> -Dan
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> On Thu, Sep 24, 2026 at 11:55 AM Antoine
> > >> Pitrou <
> > >>>>>>>>>>>> antoine@python.org>
> > >>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> Le 24/09/2026 à 20:11, Russell Spitzer a
> > >> écrit
> > >>> :
> > >>>>>>>>>>>>>>>>>> I think something I haven't seen yet is the
> > >>>> clear
> > >>>>>>>>> use-case
> > >>>>>>>>>>> for
> > >>>>>>>>>>>>>>>> non-finite
> > >>>>>>>>>>>>>>>>>> values within a vector type. There has
> > >> been a
> > >>>> lot
> > >>>>>> of
> > >>>>>>>>>>>> theoretical
> > >>>>>>>>>>>>>>>>> discussion
> > >>>>>>>>>>>>>>>>>> here about use cases but what toolings
> > >>>> currently
> > >>>>>>>> produce
> > >>>>>>>>>>> these
> > >>>>>>>>>>>>>>> vectors
> > >>>>>>>>>>>>>>>>> and
> > >>>>>>>>>>>>>>>>>> how do they function in downstream engines?
> > >>> I'd
> > >>>>>> like
> > >>>>>>> to
> > >>>>>>>>>> limit
> > >>>>>>>>>>>>>>>> introducing
> > >>>>>>>>>>>>>>>>>> types that have a wide range of values
> > >> which
> > >>>> are
> > >>>>>>> simply
> > >>>>>>>>> are
> > >>>>>>>>>>>>> failure
> > >>>>>>>>>>>>>>>> modes
> > >>>>>>>>>>>>>>>>>> of different flavors
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> "Vector" has a more general sense than ML and
> > >>>> it's
> > >>>>>>> quite
> > >>>>>>>>>>> trivial
> > >>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>> produce vectors with non-finite values using
> > >>> e.g.
> > >>>>>> NumPy
> > >>>>>>>> or
> > >>>>>>>>>>> Pandas
> > >>>>>>>>>>>>> (or
> > >>>>>>>>>>>>>>>>> any other piece of dataframe or scientific
> > >>>>> computing
> > >>>>>>>>>> software).
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> It's not like the general concept of a vector
> > >>> is
> > >>>> a
> > >>>>>> IETF
> > >>>>>>>>>>> standard
> > >>>>>>>>>>>> or
> > >>>>>>>>>>>>>>>>> anything, so analogies with JSON or UTF8
> > >> (where
> > >>>> the
> > >>>>>>>> values
> > >>>>>>>>>> are
> > >>>>>>>>>>>>>> expected
> > >>>>>>>>>>>>>>>>> to be valid according to the respective
> > >> specs)
> > >>> do
> > >>>>> not
> > >>>>>>>>> apply.
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> This discussion would IMHO have a better
> > >> chance
> > >>>> of
> > >>>>>>>>> converging
> > >>>>>>>>>>> if
> > >>>>>>>>>>>> we
> > >>>>>>>>>>>>>>>>> separated the two orthogonal concerns of:
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> 1. representing fixed-length list/vector data
> > >>> in
> > >>>>>>> Parquet
> > >>>>>>>>>> (with
> > >>>>>>>>>>> a
> > >>>>>>>>>>>>>>>>> potential limitation to non-nested types)
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> 2. representing the information that a given
> > >>>> column
> > >>>>>> (or
> > >>>>>>>>>> column
> > >>>>>>>>>>>>> chunk,
> > >>>>>>>>>>>>>>>>> etc.) does not contain non-finite values
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> ... both of which are useful on their own
> > >>> right,
> > >>>>> but
> > >>>>>> do
> > >>>>>>>> not
> > >>>>>>>>>>> have
> > >>>>>>>>>>>> to
> > >>>>>>>>>>>>>> be
> > >>>>>>>>>>>>>>>>> conflated and bundled together.
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> Regards
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> Antoine.
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>>
> > >>>>
> > >>>
> > >>
> >
> >
>

Re: [DISCUSS] Numeric Vector type: physical representation (outcome of focused call)

[Jan Finis <jp...@gmail.com>]

>
>
>
Daniel,

alright, so no physical optimizations intended.  We only want this for
semantic reasons, not for any significant physical optimization. If we're
all on the same page here, that is good 👍.

To phrase my remaining problem with this more concretely:
*Let's say we add this type. What cool new stuff can we now do that we
couldn't do before?*
That is the litmus test that a type has to pass to be useful IMHO. And here
I can't come up with a tangible use case that wasn't possible before.
Especially if we one day have fixed size lists in Parquet, the remaining
niche for this type becomes so small, that the added complexity isn't worth
the benefit IMHO.

As I guess most people on here are active maintainers of a Parquet
implementation: Concretely, what would you be using this type for if we had
it? Would you allow math only on fields tagged with this type? What if that
math yields INF or NaN? Not allowed?

Especially if we also discuss having both VECTOR and FINITE_VECTOR. How
will you guys wire this up with the type systems of your engines? Do your
engines also have two types here? Do you type-decay a FINITE_VECTOR to a
VECTOR once someone does a multiplication that could yield INF? By itself,
a FINITE_VECTOR sounds good. Holistically looked at from a type system
perspective, it comes with a lot of complexity. Do the benefits really
outweigh the complexity here? That's why I would rather have the bare
minimum: A fixed size array (list) of floats or at least just a vector
without finite. Then all these detailed questions go away.

I'll take this one step further... Why is there any debate about this at
> all? Why can't a type simply be added to the *specification* as an
> administrative detail and be done?


Andrew,

Parquet is supposed to be a lingua franca of data. The specification is the
authoritative truth for this language and all implementations are expected
to implement as much as possible from this spec, so anything in the spec
does causes implementation effort. So just adding anything to this spec
without debate is not a good idea. It is even the other way around: We need
to have way higher rigor with the spec than we need to have with any
implementations. Implementations can be changed, the spec needs to be
stable.

A tag-only type that doesn't come with any restrictions can be added with
little effort. But here we want to tie restrictions to it (no NaN/Inf), so
we would expect implementations to enforce these. So there is some
implementation effort expected here.

Also, even if we only wanted a tag type, I would still argue that you
shouldn't add those without a debate. What if A wants XML, B wants YAML, C
wants quaternions, D wants address, E wants phone number. If we just added
any types that are "just tags", Parquet logical types would just become a
trash dump of any type that anyone ever thought to be useful. So rigorous
assessment of each new type candidate is necessary.

Finally, many engines couple parquet with another type system (e.g., Arrow
or SQL). So even with a tag only type, you at least still have the
implementation burden of wiring this up with your engine's type system.

So, just because logical types are "just" logical, that doesn't mean we can
add them without any rigor.


A few naive questions:
> 1. Would a physical definition with NANs & Inf+/- allow efficient
> encodings like ALP?
> 2. If the physical representation allows values that make encodings like
> ALP impossible, can a tighter Logical type allow different encodings?
> 3. Can we RLE or binpack a vector?


The simple general answer here is that all these things do not require a
logical type. Writers can already do any such optimization if the actual
data contains no NaNs & Infs. All these physical optimizations just require
no actual NaNs/Infs to be present in a page, they don't need a guarantee
that the whole file won't contain any.

Cheers,
Jan




Am Do., 15. Okt. 2026 um 04:28 Uhr schrieb <ku...@gmail.com>:

> Hi Everyone,
>
> Glad to see a Vector type being added. I have a few use cases I want to
> pursue once we have an agreed upon definition.
>
> In regards to that definition discussion. I feel comfortable with the idea
> of the semantic meaning Dan raises though I can’t shake a slight discomfort
> (maybe FOMO) with limiting values in the physical definition.
>
> A few naive questions:
> 1. Would a physical definition with NANs & Inf+/- allow efficient
> encodings like ALP?
> 2. If the physical representation allows values that make encodings like
> ALP impossible, can a tighter Logical type allow different encodings?
> 3. Can we RLE or binpack a vector?
>
> Best Regards,
> Kurtis C. Wright
> > On Oct 14, 2026, at 12:45, Daniel Weeks <dw...@apache.org> wrote:
> >
> > Hey Jan,
> >
> > There's a lot that I disagree with respect to the framing of these
> > arguments and how we should be thinking about introducing vector as a
> > logical type.  There's a repeated tendency to let the physical
> > representation or performance drive discussion around types as opposed to
> > what it represents.
> >
> > In parquet, establishing semantic meaning is the purpose of logical
> types.
> > The logical type separates how to interpret a value from its physical
> > representation.  We can represent a timestamp with INT64 or even a string
> > value, so why have a type for it? Even though a FIXED_SIZED_LIST and
> VECTOR
> > may have the exact same representation and performance, they have
> different
> > semantic meaning.  Databases and engines across the ecosystem already
> > recognize this differentiation and have separate representations.  The
> > choice to separate these isn't because the discussion is too tedious;
> > they're fundamentally different things.
> >
> > The finite requirement is what makes a vector a reflection of what it
> > represents.  As we've discussed in the thread, non-finite values in the
> > representation violate the properties of a vector and the value ceases to
> > be a vector.  On the other hand, a 0 vector is still a vector (you can
> > argue the utility of it, but it is a valid vector).  Implementations can
> > choose what they validate or how to handle execution errors; that's an
> > implementation concern, not a requirement.  Operations performed on a
> > vector depend on the context in which they are used, but the
> representation
> > doesn't need to solve scenarios where that application doesn't make
> sense.
> >
> > The guarantees or context needed to use a vector do not need to be
> inherent
> > in the type for it to be a useful type.  We don't need the currency used
> > for decimal or whether a floating point is a normalized value as part of
> > the type to make use of those types.  This conflates the use cases with
> the
> > type system and unnecessarily attempts to draw the context into the type
> > definition.  Similarly, you don't need to capture normalization, model,
> > etc. as part of the type for it to be useful.
> >
> > So the reason we want to add a vector type (or FINITE_VECTOR) is because
> we
> > want to store vectors with the semantic meaning and requirements based on
> > what they represent.
> >
> > -Dan
> >
> >
> >> On Wed, Oct 14, 2026 at 1:22 AM Jan Finis <jp...@gmail.com> wrote:
> >>
> >> A bit late to the party, but here we go:
> >>
> >> I have the strong sentiment that the FINITE_VECTOR type as it is
> currently
> >> being discussed will not be very useful and just increase implementation
> >> complexity. The arguments for the type are all quite weak (details see
> >> below). The FINITE part doesn't seem to yield any value and is not a
> useful
> >> restriction by itself. The vector (i.e. fixed length) part should
> rather be
> >> generalized instead of making a one-off thing. In general, the existing
> >> types are already working well for uncompressed vectors. The real
> challenge
> >> that would bring real benefits is supporting quantization. This is where
> >> you get orders of magnitude improvements to storage space and processing
> >> speed.
> >>
> >> Let's first discuss the FINITE part. So far, the only arguments I have
> read
> >> for this are:
> >>
> >> 1. many other vector databases have it
> >> 2. vectors are the hot stuff right now, so Parquet better boast having
> >> "native" type support for vectors
> >> 3. by focussing on this common special case, we can make rapid progress,
> >> while a general solution would take too long to specify
> >> 4. to give a guarantee to readers that no INF/NaN/NULL will be in the
> >> vector
> >>
> >> the first two arguments are quite weak by themselves. Vector databases
> are
> >> very specialized engines. Such engines can of course be very
> opinionated on
> >> their data types. A general format like Parquet must instead weigh
> whether
> >> a special type is worth the added complexity over a more general, widely
> >> applicable type. In this case, it is IMHO not at all. And just having a
> >> type called "vector" for advertisement reasons to make Parquet appear
> more
> >> modern seems a non-goal in the first place.
> >>
> >> The third "time to market" argument is valid for systems that don't
> need a
> >> stable interface. It is a very dangerous attitude for formats like
> Parquet
> >> that must be stable long term and should have a consolidated, concise,
> >> non-redundant type system. For such long term storage formats, we need
> to
> >> take the time to build the right thing, as any shortcuts we take will
> stay
> >> as blemishes in the format forever. If I read the discussion correctly,
> >> this is actually the main argument as of now: We have found that it is
> hard
> >> to align on how a fixed size list type should look like, so we want to
> go
> >> the easier route of a specialized type. This seems like a
> >> dangerous shortcut fueled by feature recency bias. Ask yourself: Would
> we
> >> want this, if Parquet already had a fixed size list type?
> >>
> >> So finally, the only technical argument is the fourth one: "It signals
> >> semantics and ensures that no bad data lands in the format". I see two
> big
> >> problems with this:
> >>
> >> 1. Readers *still* need to test for NaN / INF because Parquet files may
> be
> >> corrupt. Nothing can stop a malicious or buggy writer from physically
> >> writing NaN or INF. *So you save no single line of code in a reader. You
> >> save no branches, as you still need to handle the INF/NaN case.* So this
> >> doesn't help with performance at all. The only difference is, that the
> >> error message changes from "found an invalid INF value in the data" to
> >> "Your parquet file seems to be corrupt". It is actually quite the
> opposite,
> >> readers and writers get more complex having to handle this new type.
> >>
> >> If you now argue that it's not about performance, it's about defined
> >> semantics and erroring out early in writers, then the second problem
> >> arises:
> >>
> >> 2. The proposed no-NaN/INF/NULL guarantee is just one of multiple
> >> guarantees that you need for sensible vector processing. If "no
> >> INF/NaN/NULL" was the only guarantee you needed, I would agree that
> >> FINITE_VECTOR is useful, but it is by no means the only guarantee you
> need,
> >> instead, depending on the distance metric, you also need the following
> >> guarantees. E.g.:
> >>
> >>   - No vector may be the 0 vector, otherwise, e.g., cosine similarity is
> >>   undefined. Cosine similarity is often used as default, so this is
> >> actually
> >>   quite important in practice.
> >>   - If we want to use a dot product, then all vectors need to guarantee
> a
> >>   length of 1. This would actually be quite useful for performance
> >> reasons.
> >>   - The vectors must come from the same model as the vector used for the
> >>   search. Otherwise, the search yields a numeric result, but that result
> >> has
> >>   no sensible meaning.
> >>
> >> So, what use is a type that just guarantees part of the things you need
> for
> >> vector search but not others? You still need additional guarantees in
> >> addition to that type. What makes INF/NaN so important in comparison to,
> >> say, "no 0 vector present", that it warrants an own type, while the
> other
> >> guarantees do not? If we really want this type as a basis for
> cross-engine
> >> sharing of vectors, then it needs to give *all* guarantees.
> >>
> >> I'm not against a specialized vector type if it really carries *all*
> >> information to be useful by itself. If it carried all that information,
> >> then a reading engine could indeed validate that only meaningful vector
> >> searches are executed. But as long as it is just a half-baked thing, it
> is
> >> not useful by itself and just a burden on each implementer and
> maintainer.
> >> If we need extra information anyway, why not just stick to a general
> type
> >> like VECTOR (without FINITE).
> >>
> >> If you're a proponent of FINITE_VECTOR, ask yourself (or tell us) what
> >> advantage you will *really* get over just a (potentially fixed size)
> List
> >> of Float? As discussed, it's not performance. It's also not a semantic
> >> marker that is sufficient by itself. It's just a small hint that allows
> >> writers to error out once they see a NaN or INF in the input, while they
> >> will still happily write a bogus 0 vector. This seems to be of zero
> >> benefit.
> >>
> >> Anecdotally, we currently handle petabytes of vectors in production in
> >> Data360. We simply store them in a List of Float, with an optimized
> R-Level
> >> reader implementation that rapidly checks that all vectors are the same
> >> length. This solution works just fine. The overhead of R-level decoding
> is
> >> barely measurable. The check is super fast and R-levels are tiny in
> >> comparison to the payload vectors, so also the added I/O is negligible.
> >> This is also why I'm so firm in my sentiment: We are running a
> production
> >> grade system processing trillions of vectors right now and see no big
> >> issues with the types that Parquet currently provides.
> >>
> >> As I don't just want to be a naysayer and point out problems with the
> >> current design, here are the directions that I would propose indeed:
> >>
> >>   - a general fixed size list type. Don't do shortcuts just because the
> >>   discussion is too tedious.
> >>   - a vector type that indeed carries enough information to be really
> >>   useful for vector search by itself. (e.g., model name / version,
> >> optional
> >>   guarantees for no 0 vector / normalized vector length). With such a
> >> type,
> >>   you could really do well defined vector searches even if the writing
> >> engine
> >>   is completely different to the reading one.
> >>   - Standardize quantizations. This will be a real game changer for
> >>   performance.
> >>
> >> Cheers,
> >> Jan
> >>
> >>
> >>
> >>
> >>
> >>
> >>> Am Mi., 14. Okt. 2026 um 03:36 Uhr schrieb Gang Wu <us...@gmail.com>:
> >>>
> >>> I think separating them into FINITE_VECTOR and FIXED_LENGTH_LIST is a
> >> good
> >>> idea, though I think renaming FINITE_VECTOR to VECTOR looks more
> natural.
> >>> Should we move forward with proposing FINITE_VECTOR/VECTOR first?
> >>>
> >>> On Tue, Oct 13, 2026 at 6:26 PM Alkis Evlogimenos via dev <
> >>> dev@parquet.apache.org> wrote:
> >>>
> >>>> I suggest we go with VECTOR for the finite vector to keep the name the
> >>> same
> >>>> as other engines/type systems. If we want a non-finite one in the
> >> future
> >>> we
> >>>> can name that something else or we add fixed size list that will serve
> >>> the
> >>>> same purpose without a special name.
> >>>>
> >>>>
> >>>> On Thu, Oct 8, 2026 at 10:20 PM Ryan Blue <bl...@apache.org> wrote:
> >>>>
> >>>>> A logical type would:
> >>>>> - Describe data for users and readers to understand what the file
> >>>> contains
> >>>>> - Signal that repetition and definition levels will reflect a fixed
> >>>> number
> >>>>> of elements, which implementations can use to optimize decoding
> >>>>> - Guarantee that there are no null elements (because the type cannot
> >> be
> >>>>> applied to a list with optional elements)
> >>>>> - [FINITE_VECTOR] Guarantee that no elements are NaN or Infinity
> >> values
> >>>> so
> >>>>> that the vectors have a finite length and are suitable for distance
> >>>>> calculations
> >>>>>
> >>>>> The finite value restriction is a restriction on writers to prevent
> >> bad
> >>>>> data from being written.
> >>>>>
> >>>>> Ignoring logical types that are not understood is a rule for
> >> readers. A
> >>>>> reader can handle vectors and pass it on to another layer. For
> >>> instance,
> >>>> a
> >>>>> Parquet reader that doesn't support geometry passes values as
> >> `binary`.
> >>>>> That binary value can be passed to a geometry UDF, even if the
> >>> underlying
> >>>>> engine and Parquet reader doesn't have support.
> >>>>>
> >>>>> A tool that doesn't understand FINITE_VECTOR can similarly pass a
> >> list
> >>> of
> >>>>> floats along to a UDF. But it is still important to the user that bad
> >>>>> values are not present in the data. In your example, bad values are
> >>>> vectors
> >>>>> that cause the centroid calculation to fail, like vectors with
> >> infinite
> >>>>> length.
> >>>>>
> >>>>> Ryan
> >>>>>
> >>>>> On Thu, Oct 8, 2026 at 12:54 PM Will Edwards via dev <
> >>>>> dev@parquet.apache.org>
> >>>>> wrote:
> >>>>>
> >>>>>> Could we clarify the use cases for a VECTOR annotation on
> >>> LIST<FLOAT>?
> >>>>>> Specifically, what does it make possible that's impossible today,
> >> and
> >>>>> what
> >>>>>> does it make easier that's currently hard?
> >>>>>>
> >>>>>> I'm struggling to find a strong answer. I'd like every engine to
> >>> offer
> >>>>>> vector functions on lists of floats without needing a new logical
> >>> type.
> >>>>>>
> >>>>>> To be clear, I'm still enthusiastic about a vector type that
> >>> describes
> >>>>> how
> >>>>>> to unpack and scale composite representations (quantised codes plus
> >>>>> scales,
> >>>>>> codebooks, etc.), in the direction ML is going. That adds
> >> information
> >>>>>> engines don't have. A label on LIST<FLOAT> doesn't: engines already
> >>>>>> understand the physical type, and many already offer vector
> >> functions
> >>>> on
> >>>>>> it. I don't see a performance benefit, and NaN/null/Inf checks are
> >>>> cheap
> >>>>>> enough that engines should do them anyway. The label also doesn't
> >> say
> >>>>> what
> >>>>>> you can do with the column. Knowing that a column holds embeddings
> >>>>> doesn't
> >>>>>> say how they can be used, just as zipcode and age are both integers
> >>> but
> >>>>>> multiplying them is meaningless.
> >>>>>>
> >>>>>> My concern is the opposite effect. We're establishing that engines
> >>>> should
> >>>>>> ignore logical types they don't understand. So a tool that doesn't
> >>> know
> >>>>>> VECTOR will read the column as LIST<FLOAT>, let the user compute
> >>>>> centroids
> >>>>>> (say), and write plain LIST<FLOAT> back out. If other engines
> >>> restrict
> >>>>>> embedding functions to VECTOR-annotated columns, that file stops
> >>>> working
> >>>>>> with them, through no fault of the user. An annotation that doesn't
> >>>>> change
> >>>>>> what the data is shouldn't be able to make it unusable.
> >>>>>>
> >>>>>> Am I missing a use case?
> >>>>>>
> >>>>>> Best, Will
> >>>>>>
> >>>>>> On Thu, 8 Oct 2026 at 19:53, Steven Wu <st...@gmail.com>
> >> wrote:
> >>>>>>
> >>>>>>> I also think a FINITE_VECTOR type brings a lot of values to many
> >>> use
> >>>>>> cases.
> >>>>>>> It is would be great to make progress on the FINITE_VECTOR, and
> >> not
> >>>> to
> >>>>>> fit
> >>>>>>> every use cases into the FIXED_LENGTH_LIST.
> >>>>>>>
> >>>>>>> On Mon, Oct 5, 2026 at 1:26 PM Ryan Blue <bl...@apache.org>
> >> wrote:
> >>>>>>>
> >>>>>>>> Going through this thread again, it looks like there are a few
> >>>> things
> >>>>>>> that
> >>>>>>>> people generally agree on:
> >>>>>>>>
> >>>>>>>> - In the short term, we want to introduce a logical type or
> >>> types;
> >>>> a
> >>>>>>> better
> >>>>>>>> physical representation can be done later. We can focus on
> >>> logical
> >>>>>> types.
> >>>>>>>> - There is precedent in logical types for restricting values,
> >>> like
> >>>>> UTF8
> >>>>>>>> strings to contain only UTF-8 codepoints. I think we also agree
> >>>> that
> >>>>> we
> >>>>>>>> could restrict vector elements to non-nested or primitive
> >> types.
> >>>>>>>> - There are many examples of databases that restrict their
> >> vector
> >>>>> types
> >>>>>>> to
> >>>>>>>> non-null, finite elements.
> >>>>>>>> - Some people want to have write-time guarantees that vectors
> >>> have
> >>>>> only
> >>>>>>>> non-null, finite elements
> >>>>>>>> - Some people want to extend this work to include tensors,
> >>> shorter
> >>>>>>> floating
> >>>>>>>> point representations, variable lengths, NaN or Infinity
> >>> elements,
> >>>>> etc.
> >>>>>>>>
> >>>>>>>> It sounds like we are nearly there, but have two halves of the
> >>>>>> community
> >>>>>>>> that care about different things. That just means we should
> >> have
> >>>>>> multiple
> >>>>>>>> types. It's entirely reasonable to want a type that allows NaN
> >>>>> values,
> >>>>>>> and
> >>>>>>>> it's also reasonable for those of us building databases, table
> >>>>>> standards,
> >>>>>>>> and vector indexes to care about guarantees that elements are
> >>>> finite.
> >>>>>>>> There's more than enough examples to show that this is a
> >>> legitimate
> >>>>> use
> >>>>>>>> case.
> >>>>>>>>
> >>>>>>>> Since I don't see the concerns of the different use cases
> >>> changing,
> >>>>> it
> >>>>>>>> looks like we need to find a compromise to move forward. I
> >>> propose
> >>>>> that
> >>>>>>> we
> >>>>>>>> introduce a FINITE_VECTOR type and separately a
> >> FIXED_LENGTH_LIST
> >>>> or
> >>>>>>>> another VECTOR type that allows infinite and NaN values.
> >>>>>>>>
> >>>>>>>> Can we agree on a compromise? Is there a strong reason to try
> >> to
> >>>> fit
> >>>>>>> these
> >>>>>>>> use cases into a single type? What if that means not making
> >>>> progress?
> >>>>>>>>
> >>>>>>>> Ryan
> >>>>>>>>
> >>>>>>>> On Mon, Oct 5, 2026 at 12:15 PM Philipp Fischbeck via dev <
> >>>>>>>> dev@parquet.apache.org> wrote:
> >>>>>>>>
> >>>>>>>>> Thinking about this more, I am also fine with pulling out the
> >>>>>> “NaN/Inf”
> >>>>>>>>> requirement. Either into a type boolean (my preference), or
> >>>> putting
> >>>>>> no
> >>>>>>>>> restrictions on the Parquet level, allowing all IEEE 754
> >> float
> >>>>>> numbers.
> >>>>>>>> If
> >>>>>>>>> some top-level application wants to put additional
> >> constraints
> >>> on
> >>>>> the
> >>>>>>>> data
> >>>>>>>>> type, like Iceberg not allowing NaN/infinity, it can still do
> >>> so
> >>>>> one
> >>>>>>>> layer
> >>>>>>>>> above. It seems like this was the biggest point of discussion
> >>>>>>> remaining.
> >>>>>>>>> All these options work for me, and we should move forward
> >> with
> >>>> this
> >>>>>>>>> proposal.
> >>>>>>>>>
> >>>>>>>>> Concerning additional element types like bf16, they can be
> >>>>> introduced
> >>>>>>> as
> >>>>>>>>> Parquet types (at least in the current LIST-based spec
> >>> proposal).
> >>>>>>>>>
> >>>>>>>>> Concerning Will’s suggestion around representing VECTOR as a
> >>>>> variable
> >>>>>>>>> length list with a MIN/MAX_LENGTH metadata restriction: I
> >> think
> >>>>> that
> >>>>>> a
> >>>>>>>>> dedicated logical VECTOR type still provides enough benefits
> >> by
> >>>>>>>> describing
> >>>>>>>>> the semantics of the data, and allowing to capture future
> >>> changes
> >>>>>>> around
> >>>>>>>>> statistics. In that case, the length is better captured as an
> >>>> exact
> >>>>>>>> length
> >>>>>>>>> for this type specifically. I’m also not sure about other use
> >>>> cases
> >>>>>> for
> >>>>>>>>> MIN_LENGTH, so introducing a new type constraint system just
> >>> for
> >>>>> the
> >>>>>>>>> MIN_LENGTH=MAX_LENGTH case for vectors doesn’t have enough
> >>> value
> >>>> in
> >>>>>> my
> >>>>>>>>> view.
> >>>>>>>>>
> >>>>>>>>> Best,
> >>>>>>>>> Philipp
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> On Thu, Oct 1, 2026 at 9:00 AM Will Edwards via dev <
> >>>>>>>>> dev@parquet.apache.org>
> >>>>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>>> +1 Sebastian, a vector type suggests usage beyond
> >> normalized
> >>>>>>> embeddings
> >>>>>>>>> and
> >>>>>>>>>> engines really aren't going to be able to elide all the
> >>> checks
> >>>>>>> anyway.
> >>>>>>>>>>
> >>>>>>>>>> Antoine, you suggested we explore metadata solutions too :)
> >>>>>>>>>>
> >>>>>>>>>> Today, a database supporting CHAR(N) or VARCHAR(n) can't
> >>>> express
> >>>>>> this
> >>>>>>>> in
> >>>>>>>>>> Parquet.  Engines can write opaque blobs they understand in
> >>> the
> >>>>>>>> metadata
> >>>>>>>>>> (e.g. Arrow does) so the extra type info round-trips, but
> >>> this
> >>>>>> isn't
> >>>>>>>>>> standardized, portable, nor interoperable.
> >>>>>>>>>>
> >>>>>>>>>> So perhaps standardizing constraint metadata or new Thrift
> >>>> fields
> >>>>>> (my
> >>>>>>>>> pref)
> >>>>>>>>>> can generically support VECTOR(N) whilst also being more
> >>>> broadly
> >>>>>>> useful
> >>>>>>>>> for
> >>>>>>>>>> other use cases and data types?
> >>>>>>>>>>
> >>>>>>>>>> Imagine metadata or Thrift fields to tag the MIN_LENGTH and
> >>>>>>> MAX_LENGTH
> >>>>>>>>> for
> >>>>>>>>>> Parquet's variable-length physcial types.  These would
> >> count
> >>>>>> logical
> >>>>>>>>> units
> >>>>>>>>>> (e.g. UTF8 code points for STRING) rather than bytes.
> >>>>>>>>>>
> >>>>>>>>>> Then one engine's VECTOR(N) can be written to Parquet as
> >>>> LIST(N)
> >>>>>> with
> >>>>>>>>>> MIN_LENGTH = MAX_LENGTH = N and then readers - even
> >> different
> >>>>>>> engines -
> >>>>>>>>> can
> >>>>>>>>>> pick up these constraints if they support doing so.  We
> >> don't
> >>>>>>> actually
> >>>>>>>>> need
> >>>>>>>>>> a logical type to express a VECTOR then.
> >>>>>>>>>>
> >>>>>>>>>> This also allows standard SQL's CHAR(N) and VARCHAR(N) to
> >> be
> >>>>>>> expressed
> >>>>>>>> on
> >>>>>>>>>> UTF8 columns etc.
> >>>>>>>>>>
> >>>>>>>>>> This also seems generally useful in the future as we
> >> approach
> >>>>>>>> expressing
> >>>>>>>>>> other tensor data shapes and types.
> >>>>>>>>>>
> >>>>>>>>>> This won't change the mechanics: the storage type doesn't
> >>>> enforce
> >>>>>>> these
> >>>>>>>>>> constraints and engines must check them. It's previously
> >>>>>> established
> >>>>>>>> that
> >>>>>>>>>> performance is just a software problem and there is no
> >>>>> fundamental
> >>>>>>>>>> performance advantage to the fixed size types in
> >> Dremel-like
> >>>>>> formats
> >>>>>>>> like
> >>>>>>>>>> Parquet.
> >>>>>>>>>>
> >>>>>>>>>> Best,
> >>>>>>>>>> Will
> >>>>>>>>>>
> >>>>>>>>>> On Wed, 30 Sept 2026 at 16:08, Sebastian Baunsgaard <
> >>>>>>>>> baunsgaard@apache.org
> >>>>>>>>>>>
> >>>>>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> After listening in on the last meeting Wednesday, I have
> >>> been
> >>>>>>>> thinking
> >>>>>>>>>>> about this quite a bit, and thought I would share my 2
> >>> cents.
> >>>>>>>>>>>
> >>>>>>>>>>> I suggest not requiring VECTOR elements to be finite, as
> >>>>> proposed
> >>>>>>> in
> >>>>>>>>> the
> >>>>>>>>>> PR
> >>>>>>>>>>> <https://github.com/apache/parquet-format/pull/624>.
> >>>>>>>>>>>
> >>>>>>>>>>> While Divjot and others note the specialization for
> >> UTF-8,
> >>>>>> Parquet
> >>>>>>>>>> already
> >>>>>>>>>>> defines DOUBLE and FLOAT as IEEE encodings
> >>>>>>>>>>> <
> >>>>>>>>
> >>>> https://github.com/apache/parquet-format/blob/master/README.md#types
> >>>>>>>>>> .
> >>>>>>>>>>>
> >>>>>>>>>>> Making NaN or +/-Inf illegal inside VECTOR would be a new
> >>>>>>> restriction
> >>>>>>>>> on
> >>>>>>>>>>> those same underlying physical types. Scalar FLOAT/DOUBLE
> >>>>> columns
> >>>>>>>> have
> >>>>>>>>>>> never had such a rule.
> >>>>>>>>>>>
> >>>>>>>>>>> It would also not match how the systems that actually
> >>> produce
> >>>>>> these
> >>>>>>>>>> vectors
> >>>>>>>>>>> treat floats. Most of them use IEEE floats, e.g. Spark,
> >>>> DuckDB,
> >>>>>>>> NumPy,
> >>>>>>>>>>> PyTorch, TensorFlow, or lower level CUDA, oneMKL, and
> >>>> OpenBLAS.
> >>>>>>>>>>>
> >>>>>>>>>>> pgvector's "all elements must be finite" rule is an
> >> insert
> >>>>>> policy.
> >>>>>>>>> While
> >>>>>>>>>> I
> >>>>>>>>>>> think Parquet as a general-purpose format should allow
> >> them
> >>>> out
> >>>>>> of
> >>>>>>>> the
> >>>>>>>>>> box.
> >>>>>>>>>>> It is a user system decision to disallow writing these
> >>> values
> >>>>> or
> >>>>>> to
> >>>>>>>>> throw
> >>>>>>>>>>> errors when encountering them.
> >>>>>>>>>>>
> >>>>>>>>>>> A type-level ban also does not keep Inf/NaN out of vector
> >>>> math.
> >>>>>>>> Finite
> >>>>>>>>>>> coordinates still overflow. For instance: magnitude =
> >>>>>> ||[FLOAT_MAX,
> >>>>>>>>>>> FLOAT_MAX]|| where every coordinate is finite overflows
> >> to
> >>>>> +Inf.
> >>>>>>>>> PyTorch
> >>>>>>>>>>> documents the same overflow: tensor([1e20, 1e20]).norm()
> >> is
> >>>> inf
> >>>>>> in
> >>>>>>>>> fp32.
> >>>>>>>>>>>
> >>>>>>>>>>> A generated MAGNITUDE(vec) column would then be Inf while
> >>> the
> >>>>>>> VECTOR
> >>>>>>>>>> itself
> >>>>>>>>>>> was required to be finite. The ban only forbids storing
> >>>>> specials
> >>>>>> in
> >>>>>>>> the
> >>>>>>>>>>> coordinates, and does not prevent even simple downstream
> >>>>>>> computations
> >>>>>>>>>> from
> >>>>>>>>>>> producing them.
> >>>>>>>>>>>
> >>>>>>>>>>> I suggest keeping VECTOR as the type and add an optional
> >>> bool
> >>>>>>>>>>> VectorType.finite (default false). Engines that want
> >>>>>> pgvector-like
> >>>>>>>>>>> rejection can require the flag = true. Engines that use
> >>>> NumPy /
> >>>>>>>>> PyTorch /
> >>>>>>>>>>> Spark arrays can keep representable IEEE values. For the
> >>> same
> >>>>>>>> reason, I
> >>>>>>>>>> do
> >>>>>>>>>>> not favor adding yet another type called FINITE_VECTOR.
> >>>>>>>>>>>
> >>>>>>>>>>> Best,
> >>>>>>>>>>> Sebastian
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> On Tue, Sep 29, 2026 at 2:03 PM Divjot Arora via dev <
> >>>>>>>>>>> dev@parquet.apache.org>
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> I agree with moving forward with the proposal and
> >>> renaming
> >>>>>> VECTOR
> >>>>>>>> to
> >>>>>>>>>>>> FINITE_VECTOR or something
> >>>>>>>>>>>> similar if others feel that better communicates its
> >>>> intent. I
> >>>>>> see
> >>>>>>>>> value
> >>>>>>>>>>> in
> >>>>>>>>>>>> having a vector type that guarantees
> >>>>>>>>>>>> its values are mathematically-correct vectors for a few
> >>>>>> reasons:
> >>>>>>>>>>>>
> >>>>>>>>>>>> 1. As others have mentioned, this is similar to the
> >>> STRING
> >>>>>>> logical
> >>>>>>>>> type
> >>>>>>>>>>>> mandating UTF-8.
> >>>>>>>>>>>> 2. It is not possible to rely on statistics for
> >> detecting
> >>>> NaN
> >>>>>> or
> >>>>>>>>> +-inf
> >>>>>>>>>>>> values as all statistics are optional.
> >>>>>>>>>>>> To my knowledge, there is no precedent for requiring
> >> that
> >>>>>> writers
> >>>>>>>>> emit
> >>>>>>>>>>>> stats for specific logical types.
> >>>>>>>>>>>> 3. Without statistics, NaN/inf values could flow
> >> through
> >>>> the
> >>>>>>> engine
> >>>>>>>>> and
> >>>>>>>>>>>> block them for specific functionality.
> >>>>>>>>>>>> This seems to be Lance's behavior. However, I don't
> >> think
> >>>>> this
> >>>>>> is
> >>>>>>>> as
> >>>>>>>>>> easy
> >>>>>>>>>>>> as it sounds and Lance has run
> >>>>>>>>>>>> into several bugs due to this ([1] [2] for examples).
> >>> Their
> >>>>>>> Python
> >>>>>>>>> [3]
> >>>>>>>>>>> and
> >>>>>>>>>>>> Rust [4] APIs default to erroring on
> >>>>>>>>>>>> vectors that contain NaN values.
> >>>>>>>>>>>>
> >>>>>>>>>>>> I agree that Parquet is a general-purpose format, but I
> >>>> don't
> >>>>>>> think
> >>>>>>>>>> this
> >>>>>>>>>>>> implies the lack of domain-specific types.
> >>>>>>>>>>>> Instead, this broadens the use cases where Parquet
> >> would
> >>> be
> >>>>>>>>> considered
> >>>>>>>>>> a
> >>>>>>>>>>>> good fit. I also fully agree with
> >>>>>>>>>>>> continuing work on a general fixed-size list type if we
> >>>>> believe
> >>>>>>>> there
> >>>>>>>>>> are
> >>>>>>>>>>>> non-vector use cases that would benefit
> >>>>>>>>>>>> from such guarantees at the storage layer.
> >>>>>>>>>>>>
> >>>>>>>>>>>> [1] https://github.com/lance-format/lance/pull/3648
> >>>>>>>>>>>> [2] https://github.com/lance-format/lance/issues/8607
> >>> (see
> >>>>>>> point 2
> >>>>>>>>> in
> >>>>>>>>>>>> "root
> >>>>>>>>>>>> cause")
> >>>>>>>>>>>> [3]
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
> https://github.com/lancedb/lancedb/blob/0327618a62483feb6dca2336549845e5887cd48f/python/python/lancedb/table.py#L4807-L4911
> >>>>>>>>>>>> [4]
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
> https://github.com/lancedb/lancedb/blob/0327618a62483feb6dca2336549845e5887cd48f/rust/lancedb/src/table/add_data.rs#L105-L110
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Mon, Sep 28, 2026 at 6:19 PM Philipp Fischbeck via
> >>> dev <
> >>>>>>>>>>>> dev@parquet.apache.org> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>> I also favor moving forward with the proposal as is.
> >> I
> >>>> see
> >>>>>>> strong
> >>>>>>>>>> value
> >>>>>>>>>>>> and
> >>>>>>>>>>>>> use cases for the numeric vector type, and I agree
> >> with
> >>>> the
> >>>>>>>>>>> restrictions
> >>>>>>>>>>>>> around NaN/Inf values and element types in general.
> >>>>> Precedent
> >>>>>>>>> exists
> >>>>>>>>>> in
> >>>>>>>>>>>>> other types that restrict which physical type values
> >>> are
> >>>>>>>> logically
> >>>>>>>>>>> valid
> >>>>>>>>>>>>> (e.g., the geospatial types needing to be
> >> WKB-encoded).
> >>>> A
> >>>>>>>> general
> >>>>>>>>>>>>> fixed-size list also has value, but I think it makes
> >>>> sense
> >>>>> to
> >>>>>>>>>> separate
> >>>>>>>>>>>> the
> >>>>>>>>>>>>> two types. The semantics are relevant, and I can see
> >>> how
> >>>>> the
> >>>>>>> two
> >>>>>>>>>> types
> >>>>>>>>>>>>> might diverge further in the future regarding
> >>> statistics
> >>>> or
> >>>>>>>>>> encodings.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> I would be okay with renaming the vector type, but it
> >>>> seems
> >>>>>>> like
> >>>>>>>>> the
> >>>>>>>>>>>> VECTOR
> >>>>>>>>>>>>> name is already used across other engines mentioned
> >> in
> >>>> this
> >>>>>>>> thread.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Best,
> >>>>>>>>>>>>> Philipp
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On Sat, Sep 26, 2026 at 4:37 PM Gang Wu <
> >>>> ustcwg@gmail.com>
> >>>>>>>> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> To separate orthogonal concerns, I feel we should
> >>> treat
> >>>>> the
> >>>>>>>>> general
> >>>>>>>>>>>>>> FixedSizeList as the underlying physical layout,
> >> and
> >>>>>> separate
> >>>>>>>> it
> >>>>>>>>>> from
> >>>>>>>>>>>> the
> >>>>>>>>>>>>>> discussion of a specialized VECTOR (or broader ML
> >>>> tensor)
> >>>>>>>> logical
> >>>>>>>>>>> type.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Currently, the immediate need is to define a
> >>>> specialized
> >>>>>>> VECTOR
> >>>>>>>>>> type
> >>>>>>>>>>> so
> >>>>>>>>>>>>>> that different engines and formats have a common
> >>>> semantic
> >>>>>>>>> contract
> >>>>>>>>>>> for
> >>>>>>>>>>>>>> interoperability.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> If the main point of contention is just the
> >>>> NaN/Inf/null
> >>>>>>>>>> limitations,
> >>>>>>>>>>>>> maybe
> >>>>>>>>>>>>>> we can resolve this by adding a qualification to
> >> the
> >>>> name
> >>>>>>> (like
> >>>>>>>>>>> Andrew
> >>>>>>>>>>>>>> suggested, e.g., VECTOR_EMBEDDING or FINITE_VECTOR)
> >>> to
> >>>>>> avoid
> >>>>>>>>>>> confusion
> >>>>>>>>>>>>> with
> >>>>>>>>>>>>>> broader ML use cases. Alternatively, we could
> >>>> introduce a
> >>>>>>>> boolean
> >>>>>>>>>>> flag
> >>>>>>>>>>>>>> (e.g., optional bool VectorType.finite) as proposed
> >>>>>> earlier.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Fri, Sep 25, 2026 at 3:52 AM Andrew Lamb <
> >>>>>>>>>> andrewlamb11@gmail.com>
> >>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> I am a little confused about the points of
> >>>> disagreement
> >>>>>>> here.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Is it that the name `VECTOR` is too general for
> >> the
> >>>>>>> specific
> >>>>>>>>> use
> >>>>>>>>>>> case
> >>>>>>>>>>>>> of
> >>>>>>>>>>>>>>> fixed length arrays with non-finite values used
> >> for
> >>>>>> vector
> >>>>>>>>>>>> embeddings?
> >>>>>>>>>>>>>> If
> >>>>>>>>>>>>>>> so, maybe we could add a qualification to the
> >> name
> >>>>>>>>>> (FINITE_VECTOR?
> >>>>>>>>>>>>>>> FEATURE_VECTOR? VECTOR_EMBEDDING?)
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> It seems like there is precedent for defining
> >>> logical
> >>>>>> types
> >>>>>>>>> that
> >>>>>>>>>>>>> specify
> >>>>>>>>>>>>>>> certain physical types and restrict their
> >> contents
> >>> to
> >>>>>> help
> >>>>>>>>>> readers
> >>>>>>>>>>>>>>> interpret meaning, for example string[1] or JSON;
> >>>>>> Defining
> >>>>>>>> one
> >>>>>>>>>> for
> >>>>>>>>>>>>>> vector
> >>>>>>>>>>>>>>> embeddings seems reasonable enough to me.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Andrew
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> [1]:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
> https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#string
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On Thu, Sep 24, 2026 at 3:34 PM Daniel Weeks <
> >>>>>>>>> dweeks@apache.org>
> >>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Well, Parquet is not a single-purpose format.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Vector stores can choose whatever policy fits
> >>>> their
> >>>>>>>> usage,
> >>>>>>>>>> but
> >>>>>>>>>>>>> we're
> >>>>>>>>>>>>>>>>> talking about a general-purpose file format
> >>> that
> >>>>> aims
> >>>>>>> to
> >>>>>>>> be
> >>>>>>>>>>>> broadly
> >>>>>>>>>>>>>>>>> applicable.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> I agree that the format is intended to be
> >> general
> >>>>>>> purpose,
> >>>>>>>>> but
> >>>>>>>>>>>> types
> >>>>>>>>>>>>>>> within
> >>>>>>>>>>>>>>>> the format define specific representation and
> >>>>> semantics
> >>>>>>> to
> >>>>>>>>>>> achieve
> >>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>> goal.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> "Vector" has a more general sense than ML and
> >>>> it's
> >>>>>>> quite
> >>>>>>>>>>> trivial
> >>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>> produce vectors with non-finite values using
> >>> e.g.
> >>>>>> NumPy
> >>>>>>>> or
> >>>>>>>>>>> Pandas
> >>>>>>>>>>>>> (or
> >>>>>>>>>>>>>>>>> any other piece of dataframe or scientific
> >>>>> computing
> >>>>>>>>>> software).
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> The justification for restricting non-finite
> >>>> values
> >>>>> is
> >>>>>>>>>> directly
> >>>>>>>>>>>>>> related
> >>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>> the semantic meaning of a vector.  Vectors have
> >>>>>> specific
> >>>>>>>>>>>>>> characteristics
> >>>>>>>>>>>>>>>> (dimension, magnitude, direction, etc).
> >>> Non-finite
> >>>>>>> values
> >>>>>>>>> are
> >>>>>>>>>>>>> invalid
> >>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>> break those characteristics.  I'd liken it to
> >>>> storing
> >>>>>>>>> non-utf8
> >>>>>>>>>>>> values
> >>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>> a
> >>>>>>>>>>>>>>>> string and still calling it a string.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> That also differentiates a vector from a fixed
> >>>> length
> >>>>>>> list,
> >>>>>>>>>> which
> >>>>>>>>>>>> is
> >>>>>>>>>>>>>> more
> >>>>>>>>>>>>>>>> general and does not carry the same semantic
> >>>> meaning.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> I also don't think we need to bundle all
> >>> similarly
> >>>>>> shaped
> >>>>>>>> use
> >>>>>>>>>>> cases
> >>>>>>>>>>>>>> into
> >>>>>>>>>>>>>>>> the same solution (e.g. flxed size list).
> >> Vector
> >>>> is
> >>>>> a
> >>>>>>> well
> >>>>>>>>>>>>> established
> >>>>>>>>>>>>>>>> data type at this point and aligns with the
> >>> general
> >>>>>>> purpose
> >>>>>>>>>>> nature
> >>>>>>>>>>>> of
> >>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>> format (many other formats and databases
> >> natively
> >>>>>> support
> >>>>>>>>>> this).
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> -Dan
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> On Thu, Sep 24, 2026 at 11:55 AM Antoine
> >> Pitrou <
> >>>>>>>>>>>> antoine@python.org>
> >>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Le 24/09/2026 à 20:11, Russell Spitzer a
> >> écrit
> >>> :
> >>>>>>>>>>>>>>>>>> I think something I haven't seen yet is the
> >>>> clear
> >>>>>>>>> use-case
> >>>>>>>>>>> for
> >>>>>>>>>>>>>>>> non-finite
> >>>>>>>>>>>>>>>>>> values within a vector type. There has
> >> been a
> >>>> lot
> >>>>>> of
> >>>>>>>>>>>> theoretical
> >>>>>>>>>>>>>>>>> discussion
> >>>>>>>>>>>>>>>>>> here about use cases but what toolings
> >>>> currently
> >>>>>>>> produce
> >>>>>>>>>>> these
> >>>>>>>>>>>>>>> vectors
> >>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>> how do they function in downstream engines?
> >>> I'd
> >>>>>> like
> >>>>>>> to
> >>>>>>>>>> limit
> >>>>>>>>>>>>>>>> introducing
> >>>>>>>>>>>>>>>>>> types that have a wide range of values
> >> which
> >>>> are
> >>>>>>> simply
> >>>>>>>>> are
> >>>>>>>>>>>>> failure
> >>>>>>>>>>>>>>>> modes
> >>>>>>>>>>>>>>>>>> of different flavors
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> "Vector" has a more general sense than ML and
> >>>> it's
> >>>>>>> quite
> >>>>>>>>>>> trivial
> >>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>> produce vectors with non-finite values using
> >>> e.g.
> >>>>>> NumPy
> >>>>>>>> or
> >>>>>>>>>>> Pandas
> >>>>>>>>>>>>> (or
> >>>>>>>>>>>>>>>>> any other piece of dataframe or scientific
> >>>>> computing
> >>>>>>>>>> software).
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> It's not like the general concept of a vector
> >>> is
> >>>> a
> >>>>>> IETF
> >>>>>>>>>>> standard
> >>>>>>>>>>>> or
> >>>>>>>>>>>>>>>>> anything, so analogies with JSON or UTF8
> >> (where
> >>>> the
> >>>>>>>> values
> >>>>>>>>>> are
> >>>>>>>>>>>>>> expected
> >>>>>>>>>>>>>>>>> to be valid according to the respective
> >> specs)
> >>> do
> >>>>> not
> >>>>>>>>> apply.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> This discussion would IMHO have a better
> >> chance
> >>>> of
> >>>>>>>>> converging
> >>>>>>>>>>> if
> >>>>>>>>>>>> we
> >>>>>>>>>>>>>>>>> separated the two orthogonal concerns of:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> 1. representing fixed-length list/vector data
> >>> in
> >>>>>>> Parquet
> >>>>>>>>>> (with
> >>>>>>>>>>> a
> >>>>>>>>>>>>>>>>> potential limitation to non-nested types)
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> 2. representing the information that a given
> >>>> column
> >>>>>> (or
> >>>>>>>>>> column
> >>>>>>>>>>>>> chunk,
> >>>>>>>>>>>>>>>>> etc.) does not contain non-finite values
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> ... both of which are useful on their own
> >>> right,
> >>>>> but
> >>>>>> do
> >>>>>>>> not
> >>>>>>>>>>> have
> >>>>>>>>>>>> to
> >>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>> conflated and bundled together.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Regards
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Antoine.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
>
>

Re: [DISCUSS] Numeric Vector type: physical representation (outcome of focused call)

[Gang Wu <us...@gmail.com>]

I feel this discussion is getting overly academic around domain-specific
requirements. As a general-purpose file format, Parquet should focus on
defining clear, foundational data properties rather than trying to capture
or enforce every application-level invariant in the type system.

If we look at existing logical types in Parquet, there is a clear precedent
for how boundaries are drawn:

1) STRING requires valid UTF-8, GEOMETRY and GEOGRAPHY require
   valid WKB, and JSON and BSON require valid documents.
2) In all of these cases, the underlying physical storage cannot physically
   prevent a buggy writer from emitting invalid bytes. The logical type
   establishes a clear baseline contract, while validation remains the
   responsibility of writer and reader implementations.
3) Higher-level domain semantics (such as whether a vector is non-zero for
   cosine similarity, normalized to unit length for dot product, or tied to
   a specific embedding model) belong to specific applications, rather than
   the Parquet logical type definition.

Defining a vector logical type with a fixed length and finite, non-null
primitive elements sets a pragmatic, well-understood boundary that aligns
with mainstream engines without over-engineering the format.

On Thu, Oct 15, 2026 at 10:28 AM <ku...@gmail.com> wrote:

> Hi Everyone,
>
> Glad to see a Vector type being added. I have a few use cases I want to
> pursue once we have an agreed upon definition.
>
> In regards to that definition discussion. I feel comfortable with the idea
> of the semantic meaning Dan raises though I can’t shake a slight discomfort
> (maybe FOMO) with limiting values in the physical definition.
>
> A few naive questions:
> 1. Would a physical definition with NANs & Inf+/- allow efficient
> encodings like ALP?
> 2. If the physical representation allows values that make encodings like
> ALP impossible, can a tighter Logical type allow different encodings?
> 3. Can we RLE or binpack a vector?
>
> Best Regards,
> Kurtis C. Wright
> > On Oct 14, 2026, at 12:45, Daniel Weeks <dw...@apache.org> wrote:
> >
> > Hey Jan,
> >
> > There's a lot that I disagree with respect to the framing of these
> > arguments and how we should be thinking about introducing vector as a
> > logical type.  There's a repeated tendency to let the physical
> > representation or performance drive discussion around types as opposed to
> > what it represents.
> >
> > In parquet, establishing semantic meaning is the purpose of logical
> types.
> > The logical type separates how to interpret a value from its physical
> > representation.  We can represent a timestamp with INT64 or even a string
> > value, so why have a type for it? Even though a FIXED_SIZED_LIST and
> VECTOR
> > may have the exact same representation and performance, they have
> different
> > semantic meaning.  Databases and engines across the ecosystem already
> > recognize this differentiation and have separate representations.  The
> > choice to separate these isn't because the discussion is too tedious;
> > they're fundamentally different things.
> >
> > The finite requirement is what makes a vector a reflection of what it
> > represents.  As we've discussed in the thread, non-finite values in the
> > representation violate the properties of a vector and the value ceases to
> > be a vector.  On the other hand, a 0 vector is still a vector (you can
> > argue the utility of it, but it is a valid vector).  Implementations can
> > choose what they validate or how to handle execution errors; that's an
> > implementation concern, not a requirement.  Operations performed on a
> > vector depend on the context in which they are used, but the
> representation
> > doesn't need to solve scenarios where that application doesn't make
> sense.
> >
> > The guarantees or context needed to use a vector do not need to be
> inherent
> > in the type for it to be a useful type.  We don't need the currency used
> > for decimal or whether a floating point is a normalized value as part of
> > the type to make use of those types.  This conflates the use cases with
> the
> > type system and unnecessarily attempts to draw the context into the type
> > definition.  Similarly, you don't need to capture normalization, model,
> > etc. as part of the type for it to be useful.
> >
> > So the reason we want to add a vector type (or FINITE_VECTOR) is because
> we
> > want to store vectors with the semantic meaning and requirements based on
> > what they represent.
> >
> > -Dan
> >
> >
> >> On Wed, Oct 14, 2026 at 1:22 AM Jan Finis <jp...@gmail.com> wrote:
> >>
> >> A bit late to the party, but here we go:
> >>
> >> I have the strong sentiment that the FINITE_VECTOR type as it is
> currently
> >> being discussed will not be very useful and just increase implementation
> >> complexity. The arguments for the type are all quite weak (details see
> >> below). The FINITE part doesn't seem to yield any value and is not a
> useful
> >> restriction by itself. The vector (i.e. fixed length) part should
> rather be
> >> generalized instead of making a one-off thing. In general, the existing
> >> types are already working well for uncompressed vectors. The real
> challenge
> >> that would bring real benefits is supporting quantization. This is where
> >> you get orders of magnitude improvements to storage space and processing
> >> speed.
> >>
> >> Let's first discuss the FINITE part. So far, the only arguments I have
> read
> >> for this are:
> >>
> >> 1. many other vector databases have it
> >> 2. vectors are the hot stuff right now, so Parquet better boast having
> >> "native" type support for vectors
> >> 3. by focussing on this common special case, we can make rapid progress,
> >> while a general solution would take too long to specify
> >> 4. to give a guarantee to readers that no INF/NaN/NULL will be in the
> >> vector
> >>
> >> the first two arguments are quite weak by themselves. Vector databases
> are
> >> very specialized engines. Such engines can of course be very
> opinionated on
> >> their data types. A general format like Parquet must instead weigh
> whether
> >> a special type is worth the added complexity over a more general, widely
> >> applicable type. In this case, it is IMHO not at all. And just having a
> >> type called "vector" for advertisement reasons to make Parquet appear
> more
> >> modern seems a non-goal in the first place.
> >>
> >> The third "time to market" argument is valid for systems that don't
> need a
> >> stable interface. It is a very dangerous attitude for formats like
> Parquet
> >> that must be stable long term and should have a consolidated, concise,
> >> non-redundant type system. For such long term storage formats, we need
> to
> >> take the time to build the right thing, as any shortcuts we take will
> stay
> >> as blemishes in the format forever. If I read the discussion correctly,
> >> this is actually the main argument as of now: We have found that it is
> hard
> >> to align on how a fixed size list type should look like, so we want to
> go
> >> the easier route of a specialized type. This seems like a
> >> dangerous shortcut fueled by feature recency bias. Ask yourself: Would
> we
> >> want this, if Parquet already had a fixed size list type?
> >>
> >> So finally, the only technical argument is the fourth one: "It signals
> >> semantics and ensures that no bad data lands in the format". I see two
> big
> >> problems with this:
> >>
> >> 1. Readers *still* need to test for NaN / INF because Parquet files may
> be
> >> corrupt. Nothing can stop a malicious or buggy writer from physically
> >> writing NaN or INF. *So you save no single line of code in a reader. You
> >> save no branches, as you still need to handle the INF/NaN case.* So this
> >> doesn't help with performance at all. The only difference is, that the
> >> error message changes from "found an invalid INF value in the data" to
> >> "Your parquet file seems to be corrupt". It is actually quite the
> opposite,
> >> readers and writers get more complex having to handle this new type.
> >>
> >> If you now argue that it's not about performance, it's about defined
> >> semantics and erroring out early in writers, then the second problem
> >> arises:
> >>
> >> 2. The proposed no-NaN/INF/NULL guarantee is just one of multiple
> >> guarantees that you need for sensible vector processing. If "no
> >> INF/NaN/NULL" was the only guarantee you needed, I would agree that
> >> FINITE_VECTOR is useful, but it is by no means the only guarantee you
> need,
> >> instead, depending on the distance metric, you also need the following
> >> guarantees. E.g.:
> >>
> >>   - No vector may be the 0 vector, otherwise, e.g., cosine similarity is
> >>   undefined. Cosine similarity is often used as default, so this is
> >> actually
> >>   quite important in practice.
> >>   - If we want to use a dot product, then all vectors need to guarantee
> a
> >>   length of 1. This would actually be quite useful for performance
> >> reasons.
> >>   - The vectors must come from the same model as the vector used for the
> >>   search. Otherwise, the search yields a numeric result, but that result
> >> has
> >>   no sensible meaning.
> >>
> >> So, what use is a type that just guarantees part of the things you need
> for
> >> vector search but not others? You still need additional guarantees in
> >> addition to that type. What makes INF/NaN so important in comparison to,
> >> say, "no 0 vector present", that it warrants an own type, while the
> other
> >> guarantees do not? If we really want this type as a basis for
> cross-engine
> >> sharing of vectors, then it needs to give *all* guarantees.
> >>
> >> I'm not against a specialized vector type if it really carries *all*
> >> information to be useful by itself. If it carried all that information,
> >> then a reading engine could indeed validate that only meaningful vector
> >> searches are executed. But as long as it is just a half-baked thing, it
> is
> >> not useful by itself and just a burden on each implementer and
> maintainer.
> >> If we need extra information anyway, why not just stick to a general
> type
> >> like VECTOR (without FINITE).
> >>
> >> If you're a proponent of FINITE_VECTOR, ask yourself (or tell us) what
> >> advantage you will *really* get over just a (potentially fixed size)
> List
> >> of Float? As discussed, it's not performance. It's also not a semantic
> >> marker that is sufficient by itself. It's just a small hint that allows
> >> writers to error out once they see a NaN or INF in the input, while they
> >> will still happily write a bogus 0 vector. This seems to be of zero
> >> benefit.
> >>
> >> Anecdotally, we currently handle petabytes of vectors in production in
> >> Data360. We simply store them in a List of Float, with an optimized
> R-Level
> >> reader implementation that rapidly checks that all vectors are the same
> >> length. This solution works just fine. The overhead of R-level decoding
> is
> >> barely measurable. The check is super fast and R-levels are tiny in
> >> comparison to the payload vectors, so also the added I/O is negligible.
> >> This is also why I'm so firm in my sentiment: We are running a
> production
> >> grade system processing trillions of vectors right now and see no big
> >> issues with the types that Parquet currently provides.
> >>
> >> As I don't just want to be a naysayer and point out problems with the
> >> current design, here are the directions that I would propose indeed:
> >>
> >>   - a general fixed size list type. Don't do shortcuts just because the
> >>   discussion is too tedious.
> >>   - a vector type that indeed carries enough information to be really
> >>   useful for vector search by itself. (e.g., model name / version,
> >> optional
> >>   guarantees for no 0 vector / normalized vector length). With such a
> >> type,
> >>   you could really do well defined vector searches even if the writing
> >> engine
> >>   is completely different to the reading one.
> >>   - Standardize quantizations. This will be a real game changer for
> >>   performance.
> >>
> >> Cheers,
> >> Jan
> >>
> >>
> >>
> >>
> >>
> >>
> >>> Am Mi., 14. Okt. 2026 um 03:36 Uhr schrieb Gang Wu <us...@gmail.com>:
> >>>
> >>> I think separating them into FINITE_VECTOR and FIXED_LENGTH_LIST is a
> >> good
> >>> idea, though I think renaming FINITE_VECTOR to VECTOR looks more
> natural.
> >>> Should we move forward with proposing FINITE_VECTOR/VECTOR first?
> >>>
> >>> On Tue, Oct 13, 2026 at 6:26 PM Alkis Evlogimenos via dev <
> >>> dev@parquet.apache.org> wrote:
> >>>
> >>>> I suggest we go with VECTOR for the finite vector to keep the name the
> >>> same
> >>>> as other engines/type systems. If we want a non-finite one in the
> >> future
> >>> we
> >>>> can name that something else or we add fixed size list that will serve
> >>> the
> >>>> same purpose without a special name.
> >>>>
> >>>>
> >>>> On Thu, Oct 8, 2026 at 10:20 PM Ryan Blue <bl...@apache.org> wrote:
> >>>>
> >>>>> A logical type would:
> >>>>> - Describe data for users and readers to understand what the file
> >>>> contains
> >>>>> - Signal that repetition and definition levels will reflect a fixed
> >>>> number
> >>>>> of elements, which implementations can use to optimize decoding
> >>>>> - Guarantee that there are no null elements (because the type cannot
> >> be
> >>>>> applied to a list with optional elements)
> >>>>> - [FINITE_VECTOR] Guarantee that no elements are NaN or Infinity
> >> values
> >>>> so
> >>>>> that the vectors have a finite length and are suitable for distance
> >>>>> calculations
> >>>>>
> >>>>> The finite value restriction is a restriction on writers to prevent
> >> bad
> >>>>> data from being written.
> >>>>>
> >>>>> Ignoring logical types that are not understood is a rule for
> >> readers. A
> >>>>> reader can handle vectors and pass it on to another layer. For
> >>> instance,
> >>>> a
> >>>>> Parquet reader that doesn't support geometry passes values as
> >> `binary`.
> >>>>> That binary value can be passed to a geometry UDF, even if the
> >>> underlying
> >>>>> engine and Parquet reader doesn't have support.
> >>>>>
> >>>>> A tool that doesn't understand FINITE_VECTOR can similarly pass a
> >> list
> >>> of
> >>>>> floats along to a UDF. But it is still important to the user that bad
> >>>>> values are not present in the data. In your example, bad values are
> >>>> vectors
> >>>>> that cause the centroid calculation to fail, like vectors with
> >> infinite
> >>>>> length.
> >>>>>
> >>>>> Ryan
> >>>>>
> >>>>> On Thu, Oct 8, 2026 at 12:54 PM Will Edwards via dev <
> >>>>> dev@parquet.apache.org>
> >>>>> wrote:
> >>>>>
> >>>>>> Could we clarify the use cases for a VECTOR annotation on
> >>> LIST<FLOAT>?
> >>>>>> Specifically, what does it make possible that's impossible today,
> >> and
> >>>>> what
> >>>>>> does it make easier that's currently hard?
> >>>>>>
> >>>>>> I'm struggling to find a strong answer. I'd like every engine to
> >>> offer
> >>>>>> vector functions on lists of floats without needing a new logical
> >>> type.
> >>>>>>
> >>>>>> To be clear, I'm still enthusiastic about a vector type that
> >>> describes
> >>>>> how
> >>>>>> to unpack and scale composite representations (quantised codes plus
> >>>>> scales,
> >>>>>> codebooks, etc.), in the direction ML is going. That adds
> >> information
> >>>>>> engines don't have. A label on LIST<FLOAT> doesn't: engines already
> >>>>>> understand the physical type, and many already offer vector
> >> functions
> >>>> on
> >>>>>> it. I don't see a performance benefit, and NaN/null/Inf checks are
> >>>> cheap
> >>>>>> enough that engines should do them anyway. The label also doesn't
> >> say
> >>>>> what
> >>>>>> you can do with the column. Knowing that a column holds embeddings
> >>>>> doesn't
> >>>>>> say how they can be used, just as zipcode and age are both integers
> >>> but
> >>>>>> multiplying them is meaningless.
> >>>>>>
> >>>>>> My concern is the opposite effect. We're establishing that engines
> >>>> should
> >>>>>> ignore logical types they don't understand. So a tool that doesn't
> >>> know
> >>>>>> VECTOR will read the column as LIST<FLOAT>, let the user compute
> >>>>> centroids
> >>>>>> (say), and write plain LIST<FLOAT> back out. If other engines
> >>> restrict
> >>>>>> embedding functions to VECTOR-annotated columns, that file stops
> >>>> working
> >>>>>> with them, through no fault of the user. An annotation that doesn't
> >>>>> change
> >>>>>> what the data is shouldn't be able to make it unusable.
> >>>>>>
> >>>>>> Am I missing a use case?
> >>>>>>
> >>>>>> Best, Will
> >>>>>>
> >>>>>> On Thu, 8 Oct 2026 at 19:53, Steven Wu <st...@gmail.com>
> >> wrote:
> >>>>>>
> >>>>>>> I also think a FINITE_VECTOR type brings a lot of values to many
> >>> use
> >>>>>> cases.
> >>>>>>> It is would be great to make progress on the FINITE_VECTOR, and
> >> not
> >>>> to
> >>>>>> fit
> >>>>>>> every use cases into the FIXED_LENGTH_LIST.
> >>>>>>>
> >>>>>>> On Mon, Oct 5, 2026 at 1:26 PM Ryan Blue <bl...@apache.org>
> >> wrote:
> >>>>>>>
> >>>>>>>> Going through this thread again, it looks like there are a few
> >>>> things
> >>>>>>> that
> >>>>>>>> people generally agree on:
> >>>>>>>>
> >>>>>>>> - In the short term, we want to introduce a logical type or
> >>> types;
> >>>> a
> >>>>>>> better
> >>>>>>>> physical representation can be done later. We can focus on
> >>> logical
> >>>>>> types.
> >>>>>>>> - There is precedent in logical types for restricting values,
> >>> like
> >>>>> UTF8
> >>>>>>>> strings to contain only UTF-8 codepoints. I think we also agree
> >>>> that
> >>>>> we
> >>>>>>>> could restrict vector elements to non-nested or primitive
> >> types.
> >>>>>>>> - There are many examples of databases that restrict their
> >> vector
> >>>>> types
> >>>>>>> to
> >>>>>>>> non-null, finite elements.
> >>>>>>>> - Some people want to have write-time guarantees that vectors
> >>> have
> >>>>> only
> >>>>>>>> non-null, finite elements
> >>>>>>>> - Some people want to extend this work to include tensors,
> >>> shorter
> >>>>>>> floating
> >>>>>>>> point representations, variable lengths, NaN or Infinity
> >>> elements,
> >>>>> etc.
> >>>>>>>>
> >>>>>>>> It sounds like we are nearly there, but have two halves of the
> >>>>>> community
> >>>>>>>> that care about different things. That just means we should
> >> have
> >>>>>> multiple
> >>>>>>>> types. It's entirely reasonable to want a type that allows NaN
> >>>>> values,
> >>>>>>> and
> >>>>>>>> it's also reasonable for those of us building databases, table
> >>>>>> standards,
> >>>>>>>> and vector indexes to care about guarantees that elements are
> >>>> finite.
> >>>>>>>> There's more than enough examples to show that this is a
> >>> legitimate
> >>>>> use
> >>>>>>>> case.
> >>>>>>>>
> >>>>>>>> Since I don't see the concerns of the different use cases
> >>> changing,
> >>>>> it
> >>>>>>>> looks like we need to find a compromise to move forward. I
> >>> propose
> >>>>> that
> >>>>>>> we
> >>>>>>>> introduce a FINITE_VECTOR type and separately a
> >> FIXED_LENGTH_LIST
> >>>> or
> >>>>>>>> another VECTOR type that allows infinite and NaN values.
> >>>>>>>>
> >>>>>>>> Can we agree on a compromise? Is there a strong reason to try
> >> to
> >>>> fit
> >>>>>>> these
> >>>>>>>> use cases into a single type? What if that means not making
> >>>> progress?
> >>>>>>>>
> >>>>>>>> Ryan
> >>>>>>>>
> >>>>>>>> On Mon, Oct 5, 2026 at 12:15 PM Philipp Fischbeck via dev <
> >>>>>>>> dev@parquet.apache.org> wrote:
> >>>>>>>>
> >>>>>>>>> Thinking about this more, I am also fine with pulling out the
> >>>>>> “NaN/Inf”
> >>>>>>>>> requirement. Either into a type boolean (my preference), or
> >>>> putting
> >>>>>> no
> >>>>>>>>> restrictions on the Parquet level, allowing all IEEE 754
> >> float
> >>>>>> numbers.
> >>>>>>>> If
> >>>>>>>>> some top-level application wants to put additional
> >> constraints
> >>> on
> >>>>> the
> >>>>>>>> data
> >>>>>>>>> type, like Iceberg not allowing NaN/infinity, it can still do
> >>> so
> >>>>> one
> >>>>>>>> layer
> >>>>>>>>> above. It seems like this was the biggest point of discussion
> >>>>>>> remaining.
> >>>>>>>>> All these options work for me, and we should move forward
> >> with
> >>>> this
> >>>>>>>>> proposal.
> >>>>>>>>>
> >>>>>>>>> Concerning additional element types like bf16, they can be
> >>>>> introduced
> >>>>>>> as
> >>>>>>>>> Parquet types (at least in the current LIST-based spec
> >>> proposal).
> >>>>>>>>>
> >>>>>>>>> Concerning Will’s suggestion around representing VECTOR as a
> >>>>> variable
> >>>>>>>>> length list with a MIN/MAX_LENGTH metadata restriction: I
> >> think
> >>>>> that
> >>>>>> a
> >>>>>>>>> dedicated logical VECTOR type still provides enough benefits
> >> by
> >>>>>>>> describing
> >>>>>>>>> the semantics of the data, and allowing to capture future
> >>> changes
> >>>>>>> around
> >>>>>>>>> statistics. In that case, the length is better captured as an
> >>>> exact
> >>>>>>>> length
> >>>>>>>>> for this type specifically. I’m also not sure about other use
> >>>> cases
> >>>>>> for
> >>>>>>>>> MIN_LENGTH, so introducing a new type constraint system just
> >>> for
> >>>>> the
> >>>>>>>>> MIN_LENGTH=MAX_LENGTH case for vectors doesn’t have enough
> >>> value
> >>>> in
> >>>>>> my
> >>>>>>>>> view.
> >>>>>>>>>
> >>>>>>>>> Best,
> >>>>>>>>> Philipp
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> On Thu, Oct 1, 2026 at 9:00 AM Will Edwards via dev <
> >>>>>>>>> dev@parquet.apache.org>
> >>>>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>>> +1 Sebastian, a vector type suggests usage beyond
> >> normalized
> >>>>>>> embeddings
> >>>>>>>>> and
> >>>>>>>>>> engines really aren't going to be able to elide all the
> >>> checks
> >>>>>>> anyway.
> >>>>>>>>>>
> >>>>>>>>>> Antoine, you suggested we explore metadata solutions too :)
> >>>>>>>>>>
> >>>>>>>>>> Today, a database supporting CHAR(N) or VARCHAR(n) can't
> >>>> express
> >>>>>> this
> >>>>>>>> in
> >>>>>>>>>> Parquet.  Engines can write opaque blobs they understand in
> >>> the
> >>>>>>>> metadata
> >>>>>>>>>> (e.g. Arrow does) so the extra type info round-trips, but
> >>> this
> >>>>>> isn't
> >>>>>>>>>> standardized, portable, nor interoperable.
> >>>>>>>>>>
> >>>>>>>>>> So perhaps standardizing constraint metadata or new Thrift
> >>>> fields
> >>>>>> (my
> >>>>>>>>> pref)
> >>>>>>>>>> can generically support VECTOR(N) whilst also being more
> >>>> broadly
> >>>>>>> useful
> >>>>>>>>> for
> >>>>>>>>>> other use cases and data types?
> >>>>>>>>>>
> >>>>>>>>>> Imagine metadata or Thrift fields to tag the MIN_LENGTH and
> >>>>>>> MAX_LENGTH
> >>>>>>>>> for
> >>>>>>>>>> Parquet's variable-length physcial types.  These would
> >> count
> >>>>>> logical
> >>>>>>>>> units
> >>>>>>>>>> (e.g. UTF8 code points for STRING) rather than bytes.
> >>>>>>>>>>
> >>>>>>>>>> Then one engine's VECTOR(N) can be written to Parquet as
> >>>> LIST(N)
> >>>>>> with
> >>>>>>>>>> MIN_LENGTH = MAX_LENGTH = N and then readers - even
> >> different
> >>>>>>> engines -
> >>>>>>>>> can
> >>>>>>>>>> pick up these constraints if they support doing so.  We
> >> don't
> >>>>>>> actually
> >>>>>>>>> need
> >>>>>>>>>> a logical type to express a VECTOR then.
> >>>>>>>>>>
> >>>>>>>>>> This also allows standard SQL's CHAR(N) and VARCHAR(N) to
> >> be
> >>>>>>> expressed
> >>>>>>>> on
> >>>>>>>>>> UTF8 columns etc.
> >>>>>>>>>>
> >>>>>>>>>> This also seems generally useful in the future as we
> >> approach
> >>>>>>>> expressing
> >>>>>>>>>> other tensor data shapes and types.
> >>>>>>>>>>
> >>>>>>>>>> This won't change the mechanics: the storage type doesn't
> >>>> enforce
> >>>>>>> these
> >>>>>>>>>> constraints and engines must check them. It's previously
> >>>>>> established
> >>>>>>>> that
> >>>>>>>>>> performance is just a software problem and there is no
> >>>>> fundamental
> >>>>>>>>>> performance advantage to the fixed size types in
> >> Dremel-like
> >>>>>> formats
> >>>>>>>> like
> >>>>>>>>>> Parquet.
> >>>>>>>>>>
> >>>>>>>>>> Best,
> >>>>>>>>>> Will
> >>>>>>>>>>
> >>>>>>>>>> On Wed, 30 Sept 2026 at 16:08, Sebastian Baunsgaard <
> >>>>>>>>> baunsgaard@apache.org
> >>>>>>>>>>>
> >>>>>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> After listening in on the last meeting Wednesday, I have
> >>> been
> >>>>>>>> thinking
> >>>>>>>>>>> about this quite a bit, and thought I would share my 2
> >>> cents.
> >>>>>>>>>>>
> >>>>>>>>>>> I suggest not requiring VECTOR elements to be finite, as
> >>>>> proposed
> >>>>>>> in
> >>>>>>>>> the
> >>>>>>>>>> PR
> >>>>>>>>>>> <https://github.com/apache/parquet-format/pull/624>.
> >>>>>>>>>>>
> >>>>>>>>>>> While Divjot and others note the specialization for
> >> UTF-8,
> >>>>>> Parquet
> >>>>>>>>>> already
> >>>>>>>>>>> defines DOUBLE and FLOAT as IEEE encodings
> >>>>>>>>>>> <
> >>>>>>>>
> >>>> https://github.com/apache/parquet-format/blob/master/README.md#types
> >>>>>>>>>> .
> >>>>>>>>>>>
> >>>>>>>>>>> Making NaN or +/-Inf illegal inside VECTOR would be a new
> >>>>>>> restriction
> >>>>>>>>> on
> >>>>>>>>>>> those same underlying physical types. Scalar FLOAT/DOUBLE
> >>>>> columns
> >>>>>>>> have
> >>>>>>>>>>> never had such a rule.
> >>>>>>>>>>>
> >>>>>>>>>>> It would also not match how the systems that actually
> >>> produce
> >>>>>> these
> >>>>>>>>>> vectors
> >>>>>>>>>>> treat floats. Most of them use IEEE floats, e.g. Spark,
> >>>> DuckDB,
> >>>>>>>> NumPy,
> >>>>>>>>>>> PyTorch, TensorFlow, or lower level CUDA, oneMKL, and
> >>>> OpenBLAS.
> >>>>>>>>>>>
> >>>>>>>>>>> pgvector's "all elements must be finite" rule is an
> >> insert
> >>>>>> policy.
> >>>>>>>>> While
> >>>>>>>>>> I
> >>>>>>>>>>> think Parquet as a general-purpose format should allow
> >> them
> >>>> out
> >>>>>> of
> >>>>>>>> the
> >>>>>>>>>> box.
> >>>>>>>>>>> It is a user system decision to disallow writing these
> >>> values
> >>>>> or
> >>>>>> to
> >>>>>>>>> throw
> >>>>>>>>>>> errors when encountering them.
> >>>>>>>>>>>
> >>>>>>>>>>> A type-level ban also does not keep Inf/NaN out of vector
> >>>> math.
> >>>>>>>> Finite
> >>>>>>>>>>> coordinates still overflow. For instance: magnitude =
> >>>>>> ||[FLOAT_MAX,
> >>>>>>>>>>> FLOAT_MAX]|| where every coordinate is finite overflows
> >> to
> >>>>> +Inf.
> >>>>>>>>> PyTorch
> >>>>>>>>>>> documents the same overflow: tensor([1e20, 1e20]).norm()
> >> is
> >>>> inf
> >>>>>> in
> >>>>>>>>> fp32.
> >>>>>>>>>>>
> >>>>>>>>>>> A generated MAGNITUDE(vec) column would then be Inf while
> >>> the
> >>>>>>> VECTOR
> >>>>>>>>>> itself
> >>>>>>>>>>> was required to be finite. The ban only forbids storing
> >>>>> specials
> >>>>>> in
> >>>>>>>> the
> >>>>>>>>>>> coordinates, and does not prevent even simple downstream
> >>>>>>> computations
> >>>>>>>>>> from
> >>>>>>>>>>> producing them.
> >>>>>>>>>>>
> >>>>>>>>>>> I suggest keeping VECTOR as the type and add an optional
> >>> bool
> >>>>>>>>>>> VectorType.finite (default false). Engines that want
> >>>>>> pgvector-like
> >>>>>>>>>>> rejection can require the flag = true. Engines that use
> >>>> NumPy /
> >>>>>>>>> PyTorch /
> >>>>>>>>>>> Spark arrays can keep representable IEEE values. For the
> >>> same
> >>>>>>>> reason, I
> >>>>>>>>>> do
> >>>>>>>>>>> not favor adding yet another type called FINITE_VECTOR.
> >>>>>>>>>>>
> >>>>>>>>>>> Best,
> >>>>>>>>>>> Sebastian
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> On Tue, Sep 29, 2026 at 2:03 PM Divjot Arora via dev <
> >>>>>>>>>>> dev@parquet.apache.org>
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> I agree with moving forward with the proposal and
> >>> renaming
> >>>>>> VECTOR
> >>>>>>>> to
> >>>>>>>>>>>> FINITE_VECTOR or something
> >>>>>>>>>>>> similar if others feel that better communicates its
> >>>> intent. I
> >>>>>> see
> >>>>>>>>> value
> >>>>>>>>>>> in
> >>>>>>>>>>>> having a vector type that guarantees
> >>>>>>>>>>>> its values are mathematically-correct vectors for a few
> >>>>>> reasons:
> >>>>>>>>>>>>
> >>>>>>>>>>>> 1. As others have mentioned, this is similar to the
> >>> STRING
> >>>>>>> logical
> >>>>>>>>> type
> >>>>>>>>>>>> mandating UTF-8.
> >>>>>>>>>>>> 2. It is not possible to rely on statistics for
> >> detecting
> >>>> NaN
> >>>>>> or
> >>>>>>>>> +-inf
> >>>>>>>>>>>> values as all statistics are optional.
> >>>>>>>>>>>> To my knowledge, there is no precedent for requiring
> >> that
> >>>>>> writers
> >>>>>>>>> emit
> >>>>>>>>>>>> stats for specific logical types.
> >>>>>>>>>>>> 3. Without statistics, NaN/inf values could flow
> >> through
> >>>> the
> >>>>>>> engine
> >>>>>>>>> and
> >>>>>>>>>>>> block them for specific functionality.
> >>>>>>>>>>>> This seems to be Lance's behavior. However, I don't
> >> think
> >>>>> this
> >>>>>> is
> >>>>>>>> as
> >>>>>>>>>> easy
> >>>>>>>>>>>> as it sounds and Lance has run
> >>>>>>>>>>>> into several bugs due to this ([1] [2] for examples).
> >>> Their
> >>>>>>> Python
> >>>>>>>>> [3]
> >>>>>>>>>>> and
> >>>>>>>>>>>> Rust [4] APIs default to erroring on
> >>>>>>>>>>>> vectors that contain NaN values.
> >>>>>>>>>>>>
> >>>>>>>>>>>> I agree that Parquet is a general-purpose format, but I
> >>>> don't
> >>>>>>> think
> >>>>>>>>>> this
> >>>>>>>>>>>> implies the lack of domain-specific types.
> >>>>>>>>>>>> Instead, this broadens the use cases where Parquet
> >> would
> >>> be
> >>>>>>>>> considered
> >>>>>>>>>> a
> >>>>>>>>>>>> good fit. I also fully agree with
> >>>>>>>>>>>> continuing work on a general fixed-size list type if we
> >>>>> believe
> >>>>>>>> there
> >>>>>>>>>> are
> >>>>>>>>>>>> non-vector use cases that would benefit
> >>>>>>>>>>>> from such guarantees at the storage layer.
> >>>>>>>>>>>>
> >>>>>>>>>>>> [1] https://github.com/lance-format/lance/pull/3648
> >>>>>>>>>>>> [2] https://github.com/lance-format/lance/issues/8607
> >>> (see
> >>>>>>> point 2
> >>>>>>>>> in
> >>>>>>>>>>>> "root
> >>>>>>>>>>>> cause")
> >>>>>>>>>>>> [3]
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
> https://github.com/lancedb/lancedb/blob/0327618a62483feb6dca2336549845e5887cd48f/python/python/lancedb/table.py#L4807-L4911
> >>>>>>>>>>>> [4]
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
> https://github.com/lancedb/lancedb/blob/0327618a62483feb6dca2336549845e5887cd48f/rust/lancedb/src/table/add_data.rs#L105-L110
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Mon, Sep 28, 2026 at 6:19 PM Philipp Fischbeck via
> >>> dev <
> >>>>>>>>>>>> dev@parquet.apache.org> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>> I also favor moving forward with the proposal as is.
> >> I
> >>>> see
> >>>>>>> strong
> >>>>>>>>>> value
> >>>>>>>>>>>> and
> >>>>>>>>>>>>> use cases for the numeric vector type, and I agree
> >> with
> >>>> the
> >>>>>>>>>>> restrictions
> >>>>>>>>>>>>> around NaN/Inf values and element types in general.
> >>>>> Precedent
> >>>>>>>>> exists
> >>>>>>>>>> in
> >>>>>>>>>>>>> other types that restrict which physical type values
> >>> are
> >>>>>>>> logically
> >>>>>>>>>>> valid
> >>>>>>>>>>>>> (e.g., the geospatial types needing to be
> >> WKB-encoded).
> >>>> A
> >>>>>>>> general
> >>>>>>>>>>>>> fixed-size list also has value, but I think it makes
> >>>> sense
> >>>>> to
> >>>>>>>>>> separate
> >>>>>>>>>>>> the
> >>>>>>>>>>>>> two types. The semantics are relevant, and I can see
> >>> how
> >>>>> the
> >>>>>>> two
> >>>>>>>>>> types
> >>>>>>>>>>>>> might diverge further in the future regarding
> >>> statistics
> >>>> or
> >>>>>>>>>> encodings.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> I would be okay with renaming the vector type, but it
> >>>> seems
> >>>>>>> like
> >>>>>>>>> the
> >>>>>>>>>>>> VECTOR
> >>>>>>>>>>>>> name is already used across other engines mentioned
> >> in
> >>>> this
> >>>>>>>> thread.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Best,
> >>>>>>>>>>>>> Philipp
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On Sat, Sep 26, 2026 at 4:37 PM Gang Wu <
> >>>> ustcwg@gmail.com>
> >>>>>>>> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> To separate orthogonal concerns, I feel we should
> >>> treat
> >>>>> the
> >>>>>>>>> general
> >>>>>>>>>>>>>> FixedSizeList as the underlying physical layout,
> >> and
> >>>>>> separate
> >>>>>>>> it
> >>>>>>>>>> from
> >>>>>>>>>>>> the
> >>>>>>>>>>>>>> discussion of a specialized VECTOR (or broader ML
> >>>> tensor)
> >>>>>>>> logical
> >>>>>>>>>>> type.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Currently, the immediate need is to define a
> >>>> specialized
> >>>>>>> VECTOR
> >>>>>>>>>> type
> >>>>>>>>>>> so
> >>>>>>>>>>>>>> that different engines and formats have a common
> >>>> semantic
> >>>>>>>>> contract
> >>>>>>>>>>> for
> >>>>>>>>>>>>>> interoperability.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> If the main point of contention is just the
> >>>> NaN/Inf/null
> >>>>>>>>>> limitations,
> >>>>>>>>>>>>> maybe
> >>>>>>>>>>>>>> we can resolve this by adding a qualification to
> >> the
> >>>> name
> >>>>>>> (like
> >>>>>>>>>>> Andrew
> >>>>>>>>>>>>>> suggested, e.g., VECTOR_EMBEDDING or FINITE_VECTOR)
> >>> to
> >>>>>> avoid
> >>>>>>>>>>> confusion
> >>>>>>>>>>>>> with
> >>>>>>>>>>>>>> broader ML use cases. Alternatively, we could
> >>>> introduce a
> >>>>>>>> boolean
> >>>>>>>>>>> flag
> >>>>>>>>>>>>>> (e.g., optional bool VectorType.finite) as proposed
> >>>>>> earlier.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Fri, Sep 25, 2026 at 3:52 AM Andrew Lamb <
> >>>>>>>>>> andrewlamb11@gmail.com>
> >>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> I am a little confused about the points of
> >>>> disagreement
> >>>>>>> here.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Is it that the name `VECTOR` is too general for
> >> the
> >>>>>>> specific
> >>>>>>>>> use
> >>>>>>>>>>> case
> >>>>>>>>>>>>> of
> >>>>>>>>>>>>>>> fixed length arrays with non-finite values used
> >> for
> >>>>>> vector
> >>>>>>>>>>>> embeddings?
> >>>>>>>>>>>>>> If
> >>>>>>>>>>>>>>> so, maybe we could add a qualification to the
> >> name
> >>>>>>>>>> (FINITE_VECTOR?
> >>>>>>>>>>>>>>> FEATURE_VECTOR? VECTOR_EMBEDDING?)
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> It seems like there is precedent for defining
> >>> logical
> >>>>>> types
> >>>>>>>>> that
> >>>>>>>>>>>>> specify
> >>>>>>>>>>>>>>> certain physical types and restrict their
> >> contents
> >>> to
> >>>>>> help
> >>>>>>>>>> readers
> >>>>>>>>>>>>>>> interpret meaning, for example string[1] or JSON;
> >>>>>> Defining
> >>>>>>>> one
> >>>>>>>>>> for
> >>>>>>>>>>>>>> vector
> >>>>>>>>>>>>>>> embeddings seems reasonable enough to me.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Andrew
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> [1]:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
> https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#string
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On Thu, Sep 24, 2026 at 3:34 PM Daniel Weeks <
> >>>>>>>>> dweeks@apache.org>
> >>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Well, Parquet is not a single-purpose format.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Vector stores can choose whatever policy fits
> >>>> their
> >>>>>>>> usage,
> >>>>>>>>>> but
> >>>>>>>>>>>>> we're
> >>>>>>>>>>>>>>>>> talking about a general-purpose file format
> >>> that
> >>>>> aims
> >>>>>>> to
> >>>>>>>> be
> >>>>>>>>>>>> broadly
> >>>>>>>>>>>>>>>>> applicable.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> I agree that the format is intended to be
> >> general
> >>>>>>> purpose,
> >>>>>>>>> but
> >>>>>>>>>>>> types
> >>>>>>>>>>>>>>> within
> >>>>>>>>>>>>>>>> the format define specific representation and
> >>>>> semantics
> >>>>>>> to
> >>>>>>>>>>> achieve
> >>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>> goal.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> "Vector" has a more general sense than ML and
> >>>> it's
> >>>>>>> quite
> >>>>>>>>>>> trivial
> >>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>> produce vectors with non-finite values using
> >>> e.g.
> >>>>>> NumPy
> >>>>>>>> or
> >>>>>>>>>>> Pandas
> >>>>>>>>>>>>> (or
> >>>>>>>>>>>>>>>>> any other piece of dataframe or scientific
> >>>>> computing
> >>>>>>>>>> software).
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> The justification for restricting non-finite
> >>>> values
> >>>>> is
> >>>>>>>>>> directly
> >>>>>>>>>>>>>> related
> >>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>> the semantic meaning of a vector.  Vectors have
> >>>>>> specific
> >>>>>>>>>>>>>> characteristics
> >>>>>>>>>>>>>>>> (dimension, magnitude, direction, etc).
> >>> Non-finite
> >>>>>>> values
> >>>>>>>>> are
> >>>>>>>>>>>>> invalid
> >>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>> break those characteristics.  I'd liken it to
> >>>> storing
> >>>>>>>>> non-utf8
> >>>>>>>>>>>> values
> >>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>> a
> >>>>>>>>>>>>>>>> string and still calling it a string.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> That also differentiates a vector from a fixed
> >>>> length
> >>>>>>> list,
> >>>>>>>>>> which
> >>>>>>>>>>>> is
> >>>>>>>>>>>>>> more
> >>>>>>>>>>>>>>>> general and does not carry the same semantic
> >>>> meaning.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> I also don't think we need to bundle all
> >>> similarly
> >>>>>> shaped
> >>>>>>>> use
> >>>>>>>>>>> cases
> >>>>>>>>>>>>>> into
> >>>>>>>>>>>>>>>> the same solution (e.g. flxed size list).
> >> Vector
> >>>> is
> >>>>> a
> >>>>>>> well
> >>>>>>>>>>>>> established
> >>>>>>>>>>>>>>>> data type at this point and aligns with the
> >>> general
> >>>>>>> purpose
> >>>>>>>>>>> nature
> >>>>>>>>>>>> of
> >>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>> format (many other formats and databases
> >> natively
> >>>>>> support
> >>>>>>>>>> this).
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> -Dan
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> On Thu, Sep 24, 2026 at 11:55 AM Antoine
> >> Pitrou <
> >>>>>>>>>>>> antoine@python.org>
> >>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Le 24/09/2026 à 20:11, Russell Spitzer a
> >> écrit
> >>> :
> >>>>>>>>>>>>>>>>>> I think something I haven't seen yet is the
> >>>> clear
> >>>>>>>>> use-case
> >>>>>>>>>>> for
> >>>>>>>>>>>>>>>> non-finite
> >>>>>>>>>>>>>>>>>> values within a vector type. There has
> >> been a
> >>>> lot
> >>>>>> of
> >>>>>>>>>>>> theoretical
> >>>>>>>>>>>>>>>>> discussion
> >>>>>>>>>>>>>>>>>> here about use cases but what toolings
> >>>> currently
> >>>>>>>> produce
> >>>>>>>>>>> these
> >>>>>>>>>>>>>>> vectors
> >>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>> how do they function in downstream engines?
> >>> I'd
> >>>>>> like
> >>>>>>> to
> >>>>>>>>>> limit
> >>>>>>>>>>>>>>>> introducing
> >>>>>>>>>>>>>>>>>> types that have a wide range of values
> >> which
> >>>> are
> >>>>>>> simply
> >>>>>>>>> are
> >>>>>>>>>>>>> failure
> >>>>>>>>>>>>>>>> modes
> >>>>>>>>>>>>>>>>>> of different flavors
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> "Vector" has a more general sense than ML and
> >>>> it's
> >>>>>>> quite
> >>>>>>>>>>> trivial
> >>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>> produce vectors with non-finite values using
> >>> e.g.
> >>>>>> NumPy
> >>>>>>>> or
> >>>>>>>>>>> Pandas
> >>>>>>>>>>>>> (or
> >>>>>>>>>>>>>>>>> any other piece of dataframe or scientific
> >>>>> computing
> >>>>>>>>>> software).
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> It's not like the general concept of a vector
> >>> is
> >>>> a
> >>>>>> IETF
> >>>>>>>>>>> standard
> >>>>>>>>>>>> or
> >>>>>>>>>>>>>>>>> anything, so analogies with JSON or UTF8
> >> (where
> >>>> the
> >>>>>>>> values
> >>>>>>>>>> are
> >>>>>>>>>>>>>> expected
> >>>>>>>>>>>>>>>>> to be valid according to the respective
> >> specs)
> >>> do
> >>>>> not
> >>>>>>>>> apply.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> This discussion would IMHO have a better
> >> chance
> >>>> of
> >>>>>>>>> converging
> >>>>>>>>>>> if
> >>>>>>>>>>>> we
> >>>>>>>>>>>>>>>>> separated the two orthogonal concerns of:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> 1. representing fixed-length list/vector data
> >>> in
> >>>>>>> Parquet
> >>>>>>>>>> (with
> >>>>>>>>>>> a
> >>>>>>>>>>>>>>>>> potential limitation to non-nested types)
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> 2. representing the information that a given
> >>>> column
> >>>>>> (or
> >>>>>>>>>> column
> >>>>>>>>>>>>> chunk,
> >>>>>>>>>>>>>>>>> etc.) does not contain non-finite values
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> ... both of which are useful on their own
> >>> right,
> >>>>> but
> >>>>>> do
> >>>>>>>> not
> >>>>>>>>>>> have
> >>>>>>>>>>>> to
> >>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>> conflated and bundled together.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Regards
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Antoine.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
>
>

Re: [DISCUSS] Numeric Vector type: physical representation (outcome of focused call)

[ku...@gmail.com]

Hi Everyone,

Glad to see a Vector type being added. I have a few use cases I want to pursue once we have an agreed upon definition.

In regards to that definition discussion. I feel comfortable with the idea of the semantic meaning Dan raises though I can’t shake a slight discomfort (maybe FOMO) with limiting values in the physical definition.

A few naive questions:
1. Would a physical definition with NANs & Inf+/- allow efficient encodings like ALP?
2. If the physical representation allows values that make encodings like ALP impossible, can a tighter Logical type allow different encodings?
3. Can we RLE or binpack a vector?

Best Regards,
Kurtis C. Wright
> On Oct 14, 2026, at 12:45, Daniel Weeks <dw...@apache.org> wrote:
> 
> Hey Jan,
> 
> There's a lot that I disagree with respect to the framing of these
> arguments and how we should be thinking about introducing vector as a
> logical type.  There's a repeated tendency to let the physical
> representation or performance drive discussion around types as opposed to
> what it represents.
> 
> In parquet, establishing semantic meaning is the purpose of logical types.
> The logical type separates how to interpret a value from its physical
> representation.  We can represent a timestamp with INT64 or even a string
> value, so why have a type for it? Even though a FIXED_SIZED_LIST and VECTOR
> may have the exact same representation and performance, they have different
> semantic meaning.  Databases and engines across the ecosystem already
> recognize this differentiation and have separate representations.  The
> choice to separate these isn't because the discussion is too tedious;
> they're fundamentally different things.
> 
> The finite requirement is what makes a vector a reflection of what it
> represents.  As we've discussed in the thread, non-finite values in the
> representation violate the properties of a vector and the value ceases to
> be a vector.  On the other hand, a 0 vector is still a vector (you can
> argue the utility of it, but it is a valid vector).  Implementations can
> choose what they validate or how to handle execution errors; that's an
> implementation concern, not a requirement.  Operations performed on a
> vector depend on the context in which they are used, but the representation
> doesn't need to solve scenarios where that application doesn't make sense.
> 
> The guarantees or context needed to use a vector do not need to be inherent
> in the type for it to be a useful type.  We don't need the currency used
> for decimal or whether a floating point is a normalized value as part of
> the type to make use of those types.  This conflates the use cases with the
> type system and unnecessarily attempts to draw the context into the type
> definition.  Similarly, you don't need to capture normalization, model,
> etc. as part of the type for it to be useful.
> 
> So the reason we want to add a vector type (or FINITE_VECTOR) is because we
> want to store vectors with the semantic meaning and requirements based on
> what they represent.
> 
> -Dan
> 
> 
>> On Wed, Oct 14, 2026 at 1:22 AM Jan Finis <jp...@gmail.com> wrote:
>> 
>> A bit late to the party, but here we go:
>> 
>> I have the strong sentiment that the FINITE_VECTOR type as it is currently
>> being discussed will not be very useful and just increase implementation
>> complexity. The arguments for the type are all quite weak (details see
>> below). The FINITE part doesn't seem to yield any value and is not a useful
>> restriction by itself. The vector (i.e. fixed length) part should rather be
>> generalized instead of making a one-off thing. In general, the existing
>> types are already working well for uncompressed vectors. The real challenge
>> that would bring real benefits is supporting quantization. This is where
>> you get orders of magnitude improvements to storage space and processing
>> speed.
>> 
>> Let's first discuss the FINITE part. So far, the only arguments I have read
>> for this are:
>> 
>> 1. many other vector databases have it
>> 2. vectors are the hot stuff right now, so Parquet better boast having
>> "native" type support for vectors
>> 3. by focussing on this common special case, we can make rapid progress,
>> while a general solution would take too long to specify
>> 4. to give a guarantee to readers that no INF/NaN/NULL will be in the
>> vector
>> 
>> the first two arguments are quite weak by themselves. Vector databases are
>> very specialized engines. Such engines can of course be very opinionated on
>> their data types. A general format like Parquet must instead weigh whether
>> a special type is worth the added complexity over a more general, widely
>> applicable type. In this case, it is IMHO not at all. And just having a
>> type called "vector" for advertisement reasons to make Parquet appear more
>> modern seems a non-goal in the first place.
>> 
>> The third "time to market" argument is valid for systems that don't need a
>> stable interface. It is a very dangerous attitude for formats like Parquet
>> that must be stable long term and should have a consolidated, concise,
>> non-redundant type system. For such long term storage formats, we need to
>> take the time to build the right thing, as any shortcuts we take will stay
>> as blemishes in the format forever. If I read the discussion correctly,
>> this is actually the main argument as of now: We have found that it is hard
>> to align on how a fixed size list type should look like, so we want to go
>> the easier route of a specialized type. This seems like a
>> dangerous shortcut fueled by feature recency bias. Ask yourself: Would we
>> want this, if Parquet already had a fixed size list type?
>> 
>> So finally, the only technical argument is the fourth one: "It signals
>> semantics and ensures that no bad data lands in the format". I see two big
>> problems with this:
>> 
>> 1. Readers *still* need to test for NaN / INF because Parquet files may be
>> corrupt. Nothing can stop a malicious or buggy writer from physically
>> writing NaN or INF. *So you save no single line of code in a reader. You
>> save no branches, as you still need to handle the INF/NaN case.* So this
>> doesn't help with performance at all. The only difference is, that the
>> error message changes from "found an invalid INF value in the data" to
>> "Your parquet file seems to be corrupt". It is actually quite the opposite,
>> readers and writers get more complex having to handle this new type.
>> 
>> If you now argue that it's not about performance, it's about defined
>> semantics and erroring out early in writers, then the second problem
>> arises:
>> 
>> 2. The proposed no-NaN/INF/NULL guarantee is just one of multiple
>> guarantees that you need for sensible vector processing. If "no
>> INF/NaN/NULL" was the only guarantee you needed, I would agree that
>> FINITE_VECTOR is useful, but it is by no means the only guarantee you need,
>> instead, depending on the distance metric, you also need the following
>> guarantees. E.g.:
>> 
>>   - No vector may be the 0 vector, otherwise, e.g., cosine similarity is
>>   undefined. Cosine similarity is often used as default, so this is
>> actually
>>   quite important in practice.
>>   - If we want to use a dot product, then all vectors need to guarantee a
>>   length of 1. This would actually be quite useful for performance
>> reasons.
>>   - The vectors must come from the same model as the vector used for the
>>   search. Otherwise, the search yields a numeric result, but that result
>> has
>>   no sensible meaning.
>> 
>> So, what use is a type that just guarantees part of the things you need for
>> vector search but not others? You still need additional guarantees in
>> addition to that type. What makes INF/NaN so important in comparison to,
>> say, "no 0 vector present", that it warrants an own type, while the other
>> guarantees do not? If we really want this type as a basis for cross-engine
>> sharing of vectors, then it needs to give *all* guarantees.
>> 
>> I'm not against a specialized vector type if it really carries *all*
>> information to be useful by itself. If it carried all that information,
>> then a reading engine could indeed validate that only meaningful vector
>> searches are executed. But as long as it is just a half-baked thing, it is
>> not useful by itself and just a burden on each implementer and maintainer.
>> If we need extra information anyway, why not just stick to a general type
>> like VECTOR (without FINITE).
>> 
>> If you're a proponent of FINITE_VECTOR, ask yourself (or tell us) what
>> advantage you will *really* get over just a (potentially fixed size) List
>> of Float? As discussed, it's not performance. It's also not a semantic
>> marker that is sufficient by itself. It's just a small hint that allows
>> writers to error out once they see a NaN or INF in the input, while they
>> will still happily write a bogus 0 vector. This seems to be of zero
>> benefit.
>> 
>> Anecdotally, we currently handle petabytes of vectors in production in
>> Data360. We simply store them in a List of Float, with an optimized R-Level
>> reader implementation that rapidly checks that all vectors are the same
>> length. This solution works just fine. The overhead of R-level decoding is
>> barely measurable. The check is super fast and R-levels are tiny in
>> comparison to the payload vectors, so also the added I/O is negligible.
>> This is also why I'm so firm in my sentiment: We are running a production
>> grade system processing trillions of vectors right now and see no big
>> issues with the types that Parquet currently provides.
>> 
>> As I don't just want to be a naysayer and point out problems with the
>> current design, here are the directions that I would propose indeed:
>> 
>>   - a general fixed size list type. Don't do shortcuts just because the
>>   discussion is too tedious.
>>   - a vector type that indeed carries enough information to be really
>>   useful for vector search by itself. (e.g., model name / version,
>> optional
>>   guarantees for no 0 vector / normalized vector length). With such a
>> type,
>>   you could really do well defined vector searches even if the writing
>> engine
>>   is completely different to the reading one.
>>   - Standardize quantizations. This will be a real game changer for
>>   performance.
>> 
>> Cheers,
>> Jan
>> 
>> 
>> 
>> 
>> 
>> 
>>> Am Mi., 14. Okt. 2026 um 03:36 Uhr schrieb Gang Wu <us...@gmail.com>:
>>> 
>>> I think separating them into FINITE_VECTOR and FIXED_LENGTH_LIST is a
>> good
>>> idea, though I think renaming FINITE_VECTOR to VECTOR looks more natural.
>>> Should we move forward with proposing FINITE_VECTOR/VECTOR first?
>>> 
>>> On Tue, Oct 13, 2026 at 6:26 PM Alkis Evlogimenos via dev <
>>> dev@parquet.apache.org> wrote:
>>> 
>>>> I suggest we go with VECTOR for the finite vector to keep the name the
>>> same
>>>> as other engines/type systems. If we want a non-finite one in the
>> future
>>> we
>>>> can name that something else or we add fixed size list that will serve
>>> the
>>>> same purpose without a special name.
>>>> 
>>>> 
>>>> On Thu, Oct 8, 2026 at 10:20 PM Ryan Blue <bl...@apache.org> wrote:
>>>> 
>>>>> A logical type would:
>>>>> - Describe data for users and readers to understand what the file
>>>> contains
>>>>> - Signal that repetition and definition levels will reflect a fixed
>>>> number
>>>>> of elements, which implementations can use to optimize decoding
>>>>> - Guarantee that there are no null elements (because the type cannot
>> be
>>>>> applied to a list with optional elements)
>>>>> - [FINITE_VECTOR] Guarantee that no elements are NaN or Infinity
>> values
>>>> so
>>>>> that the vectors have a finite length and are suitable for distance
>>>>> calculations
>>>>> 
>>>>> The finite value restriction is a restriction on writers to prevent
>> bad
>>>>> data from being written.
>>>>> 
>>>>> Ignoring logical types that are not understood is a rule for
>> readers. A
>>>>> reader can handle vectors and pass it on to another layer. For
>>> instance,
>>>> a
>>>>> Parquet reader that doesn't support geometry passes values as
>> `binary`.
>>>>> That binary value can be passed to a geometry UDF, even if the
>>> underlying
>>>>> engine and Parquet reader doesn't have support.
>>>>> 
>>>>> A tool that doesn't understand FINITE_VECTOR can similarly pass a
>> list
>>> of
>>>>> floats along to a UDF. But it is still important to the user that bad
>>>>> values are not present in the data. In your example, bad values are
>>>> vectors
>>>>> that cause the centroid calculation to fail, like vectors with
>> infinite
>>>>> length.
>>>>> 
>>>>> Ryan
>>>>> 
>>>>> On Thu, Oct 8, 2026 at 12:54 PM Will Edwards via dev <
>>>>> dev@parquet.apache.org>
>>>>> wrote:
>>>>> 
>>>>>> Could we clarify the use cases for a VECTOR annotation on
>>> LIST<FLOAT>?
>>>>>> Specifically, what does it make possible that's impossible today,
>> and
>>>>> what
>>>>>> does it make easier that's currently hard?
>>>>>> 
>>>>>> I'm struggling to find a strong answer. I'd like every engine to
>>> offer
>>>>>> vector functions on lists of floats without needing a new logical
>>> type.
>>>>>> 
>>>>>> To be clear, I'm still enthusiastic about a vector type that
>>> describes
>>>>> how
>>>>>> to unpack and scale composite representations (quantised codes plus
>>>>> scales,
>>>>>> codebooks, etc.), in the direction ML is going. That adds
>> information
>>>>>> engines don't have. A label on LIST<FLOAT> doesn't: engines already
>>>>>> understand the physical type, and many already offer vector
>> functions
>>>> on
>>>>>> it. I don't see a performance benefit, and NaN/null/Inf checks are
>>>> cheap
>>>>>> enough that engines should do them anyway. The label also doesn't
>> say
>>>>> what
>>>>>> you can do with the column. Knowing that a column holds embeddings
>>>>> doesn't
>>>>>> say how they can be used, just as zipcode and age are both integers
>>> but
>>>>>> multiplying them is meaningless.
>>>>>> 
>>>>>> My concern is the opposite effect. We're establishing that engines
>>>> should
>>>>>> ignore logical types they don't understand. So a tool that doesn't
>>> know
>>>>>> VECTOR will read the column as LIST<FLOAT>, let the user compute
>>>>> centroids
>>>>>> (say), and write plain LIST<FLOAT> back out. If other engines
>>> restrict
>>>>>> embedding functions to VECTOR-annotated columns, that file stops
>>>> working
>>>>>> with them, through no fault of the user. An annotation that doesn't
>>>>> change
>>>>>> what the data is shouldn't be able to make it unusable.
>>>>>> 
>>>>>> Am I missing a use case?
>>>>>> 
>>>>>> Best, Will
>>>>>> 
>>>>>> On Thu, 8 Oct 2026 at 19:53, Steven Wu <st...@gmail.com>
>> wrote:
>>>>>> 
>>>>>>> I also think a FINITE_VECTOR type brings a lot of values to many
>>> use
>>>>>> cases.
>>>>>>> It is would be great to make progress on the FINITE_VECTOR, and
>> not
>>>> to
>>>>>> fit
>>>>>>> every use cases into the FIXED_LENGTH_LIST.
>>>>>>> 
>>>>>>> On Mon, Oct 5, 2026 at 1:26 PM Ryan Blue <bl...@apache.org>
>> wrote:
>>>>>>> 
>>>>>>>> Going through this thread again, it looks like there are a few
>>>> things
>>>>>>> that
>>>>>>>> people generally agree on:
>>>>>>>> 
>>>>>>>> - In the short term, we want to introduce a logical type or
>>> types;
>>>> a
>>>>>>> better
>>>>>>>> physical representation can be done later. We can focus on
>>> logical
>>>>>> types.
>>>>>>>> - There is precedent in logical types for restricting values,
>>> like
>>>>> UTF8
>>>>>>>> strings to contain only UTF-8 codepoints. I think we also agree
>>>> that
>>>>> we
>>>>>>>> could restrict vector elements to non-nested or primitive
>> types.
>>>>>>>> - There are many examples of databases that restrict their
>> vector
>>>>> types
>>>>>>> to
>>>>>>>> non-null, finite elements.
>>>>>>>> - Some people want to have write-time guarantees that vectors
>>> have
>>>>> only
>>>>>>>> non-null, finite elements
>>>>>>>> - Some people want to extend this work to include tensors,
>>> shorter
>>>>>>> floating
>>>>>>>> point representations, variable lengths, NaN or Infinity
>>> elements,
>>>>> etc.
>>>>>>>> 
>>>>>>>> It sounds like we are nearly there, but have two halves of the
>>>>>> community
>>>>>>>> that care about different things. That just means we should
>> have
>>>>>> multiple
>>>>>>>> types. It's entirely reasonable to want a type that allows NaN
>>>>> values,
>>>>>>> and
>>>>>>>> it's also reasonable for those of us building databases, table
>>>>>> standards,
>>>>>>>> and vector indexes to care about guarantees that elements are
>>>> finite.
>>>>>>>> There's more than enough examples to show that this is a
>>> legitimate
>>>>> use
>>>>>>>> case.
>>>>>>>> 
>>>>>>>> Since I don't see the concerns of the different use cases
>>> changing,
>>>>> it
>>>>>>>> looks like we need to find a compromise to move forward. I
>>> propose
>>>>> that
>>>>>>> we
>>>>>>>> introduce a FINITE_VECTOR type and separately a
>> FIXED_LENGTH_LIST
>>>> or
>>>>>>>> another VECTOR type that allows infinite and NaN values.
>>>>>>>> 
>>>>>>>> Can we agree on a compromise? Is there a strong reason to try
>> to
>>>> fit
>>>>>>> these
>>>>>>>> use cases into a single type? What if that means not making
>>>> progress?
>>>>>>>> 
>>>>>>>> Ryan
>>>>>>>> 
>>>>>>>> On Mon, Oct 5, 2026 at 12:15 PM Philipp Fischbeck via dev <
>>>>>>>> dev@parquet.apache.org> wrote:
>>>>>>>> 
>>>>>>>>> Thinking about this more, I am also fine with pulling out the
>>>>>> “NaN/Inf”
>>>>>>>>> requirement. Either into a type boolean (my preference), or
>>>> putting
>>>>>> no
>>>>>>>>> restrictions on the Parquet level, allowing all IEEE 754
>> float
>>>>>> numbers.
>>>>>>>> If
>>>>>>>>> some top-level application wants to put additional
>> constraints
>>> on
>>>>> the
>>>>>>>> data
>>>>>>>>> type, like Iceberg not allowing NaN/infinity, it can still do
>>> so
>>>>> one
>>>>>>>> layer
>>>>>>>>> above. It seems like this was the biggest point of discussion
>>>>>>> remaining.
>>>>>>>>> All these options work for me, and we should move forward
>> with
>>>> this
>>>>>>>>> proposal.
>>>>>>>>> 
>>>>>>>>> Concerning additional element types like bf16, they can be
>>>>> introduced
>>>>>>> as
>>>>>>>>> Parquet types (at least in the current LIST-based spec
>>> proposal).
>>>>>>>>> 
>>>>>>>>> Concerning Will’s suggestion around representing VECTOR as a
>>>>> variable
>>>>>>>>> length list with a MIN/MAX_LENGTH metadata restriction: I
>> think
>>>>> that
>>>>>> a
>>>>>>>>> dedicated logical VECTOR type still provides enough benefits
>> by
>>>>>>>> describing
>>>>>>>>> the semantics of the data, and allowing to capture future
>>> changes
>>>>>>> around
>>>>>>>>> statistics. In that case, the length is better captured as an
>>>> exact
>>>>>>>> length
>>>>>>>>> for this type specifically. I’m also not sure about other use
>>>> cases
>>>>>> for
>>>>>>>>> MIN_LENGTH, so introducing a new type constraint system just
>>> for
>>>>> the
>>>>>>>>> MIN_LENGTH=MAX_LENGTH case for vectors doesn’t have enough
>>> value
>>>> in
>>>>>> my
>>>>>>>>> view.
>>>>>>>>> 
>>>>>>>>> Best,
>>>>>>>>> Philipp
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> On Thu, Oct 1, 2026 at 9:00 AM Will Edwards via dev <
>>>>>>>>> dev@parquet.apache.org>
>>>>>>>>> wrote:
>>>>>>>>> 
>>>>>>>>>> +1 Sebastian, a vector type suggests usage beyond
>> normalized
>>>>>>> embeddings
>>>>>>>>> and
>>>>>>>>>> engines really aren't going to be able to elide all the
>>> checks
>>>>>>> anyway.
>>>>>>>>>> 
>>>>>>>>>> Antoine, you suggested we explore metadata solutions too :)
>>>>>>>>>> 
>>>>>>>>>> Today, a database supporting CHAR(N) or VARCHAR(n) can't
>>>> express
>>>>>> this
>>>>>>>> in
>>>>>>>>>> Parquet.  Engines can write opaque blobs they understand in
>>> the
>>>>>>>> metadata
>>>>>>>>>> (e.g. Arrow does) so the extra type info round-trips, but
>>> this
>>>>>> isn't
>>>>>>>>>> standardized, portable, nor interoperable.
>>>>>>>>>> 
>>>>>>>>>> So perhaps standardizing constraint metadata or new Thrift
>>>> fields
>>>>>> (my
>>>>>>>>> pref)
>>>>>>>>>> can generically support VECTOR(N) whilst also being more
>>>> broadly
>>>>>>> useful
>>>>>>>>> for
>>>>>>>>>> other use cases and data types?
>>>>>>>>>> 
>>>>>>>>>> Imagine metadata or Thrift fields to tag the MIN_LENGTH and
>>>>>>> MAX_LENGTH
>>>>>>>>> for
>>>>>>>>>> Parquet's variable-length physcial types.  These would
>> count
>>>>>> logical
>>>>>>>>> units
>>>>>>>>>> (e.g. UTF8 code points for STRING) rather than bytes.
>>>>>>>>>> 
>>>>>>>>>> Then one engine's VECTOR(N) can be written to Parquet as
>>>> LIST(N)
>>>>>> with
>>>>>>>>>> MIN_LENGTH = MAX_LENGTH = N and then readers - even
>> different
>>>>>>> engines -
>>>>>>>>> can
>>>>>>>>>> pick up these constraints if they support doing so.  We
>> don't
>>>>>>> actually
>>>>>>>>> need
>>>>>>>>>> a logical type to express a VECTOR then.
>>>>>>>>>> 
>>>>>>>>>> This also allows standard SQL's CHAR(N) and VARCHAR(N) to
>> be
>>>>>>> expressed
>>>>>>>> on
>>>>>>>>>> UTF8 columns etc.
>>>>>>>>>> 
>>>>>>>>>> This also seems generally useful in the future as we
>> approach
>>>>>>>> expressing
>>>>>>>>>> other tensor data shapes and types.
>>>>>>>>>> 
>>>>>>>>>> This won't change the mechanics: the storage type doesn't
>>>> enforce
>>>>>>> these
>>>>>>>>>> constraints and engines must check them. It's previously
>>>>>> established
>>>>>>>> that
>>>>>>>>>> performance is just a software problem and there is no
>>>>> fundamental
>>>>>>>>>> performance advantage to the fixed size types in
>> Dremel-like
>>>>>> formats
>>>>>>>> like
>>>>>>>>>> Parquet.
>>>>>>>>>> 
>>>>>>>>>> Best,
>>>>>>>>>> Will
>>>>>>>>>> 
>>>>>>>>>> On Wed, 30 Sept 2026 at 16:08, Sebastian Baunsgaard <
>>>>>>>>> baunsgaard@apache.org
>>>>>>>>>>> 
>>>>>>>>>> wrote:
>>>>>>>>>> 
>>>>>>>>>>> After listening in on the last meeting Wednesday, I have
>>> been
>>>>>>>> thinking
>>>>>>>>>>> about this quite a bit, and thought I would share my 2
>>> cents.
>>>>>>>>>>> 
>>>>>>>>>>> I suggest not requiring VECTOR elements to be finite, as
>>>>> proposed
>>>>>>> in
>>>>>>>>> the
>>>>>>>>>> PR
>>>>>>>>>>> <https://github.com/apache/parquet-format/pull/624>.
>>>>>>>>>>> 
>>>>>>>>>>> While Divjot and others note the specialization for
>> UTF-8,
>>>>>> Parquet
>>>>>>>>>> already
>>>>>>>>>>> defines DOUBLE and FLOAT as IEEE encodings
>>>>>>>>>>> <
>>>>>>>> 
>>>> https://github.com/apache/parquet-format/blob/master/README.md#types
>>>>>>>>>> .
>>>>>>>>>>> 
>>>>>>>>>>> Making NaN or +/-Inf illegal inside VECTOR would be a new
>>>>>>> restriction
>>>>>>>>> on
>>>>>>>>>>> those same underlying physical types. Scalar FLOAT/DOUBLE
>>>>> columns
>>>>>>>> have
>>>>>>>>>>> never had such a rule.
>>>>>>>>>>> 
>>>>>>>>>>> It would also not match how the systems that actually
>>> produce
>>>>>> these
>>>>>>>>>> vectors
>>>>>>>>>>> treat floats. Most of them use IEEE floats, e.g. Spark,
>>>> DuckDB,
>>>>>>>> NumPy,
>>>>>>>>>>> PyTorch, TensorFlow, or lower level CUDA, oneMKL, and
>>>> OpenBLAS.
>>>>>>>>>>> 
>>>>>>>>>>> pgvector's "all elements must be finite" rule is an
>> insert
>>>>>> policy.
>>>>>>>>> While
>>>>>>>>>> I
>>>>>>>>>>> think Parquet as a general-purpose format should allow
>> them
>>>> out
>>>>>> of
>>>>>>>> the
>>>>>>>>>> box.
>>>>>>>>>>> It is a user system decision to disallow writing these
>>> values
>>>>> or
>>>>>> to
>>>>>>>>> throw
>>>>>>>>>>> errors when encountering them.
>>>>>>>>>>> 
>>>>>>>>>>> A type-level ban also does not keep Inf/NaN out of vector
>>>> math.
>>>>>>>> Finite
>>>>>>>>>>> coordinates still overflow. For instance: magnitude =
>>>>>> ||[FLOAT_MAX,
>>>>>>>>>>> FLOAT_MAX]|| where every coordinate is finite overflows
>> to
>>>>> +Inf.
>>>>>>>>> PyTorch
>>>>>>>>>>> documents the same overflow: tensor([1e20, 1e20]).norm()
>> is
>>>> inf
>>>>>> in
>>>>>>>>> fp32.
>>>>>>>>>>> 
>>>>>>>>>>> A generated MAGNITUDE(vec) column would then be Inf while
>>> the
>>>>>>> VECTOR
>>>>>>>>>> itself
>>>>>>>>>>> was required to be finite. The ban only forbids storing
>>>>> specials
>>>>>> in
>>>>>>>> the
>>>>>>>>>>> coordinates, and does not prevent even simple downstream
>>>>>>> computations
>>>>>>>>>> from
>>>>>>>>>>> producing them.
>>>>>>>>>>> 
>>>>>>>>>>> I suggest keeping VECTOR as the type and add an optional
>>> bool
>>>>>>>>>>> VectorType.finite (default false). Engines that want
>>>>>> pgvector-like
>>>>>>>>>>> rejection can require the flag = true. Engines that use
>>>> NumPy /
>>>>>>>>> PyTorch /
>>>>>>>>>>> Spark arrays can keep representable IEEE values. For the
>>> same
>>>>>>>> reason, I
>>>>>>>>>> do
>>>>>>>>>>> not favor adding yet another type called FINITE_VECTOR.
>>>>>>>>>>> 
>>>>>>>>>>> Best,
>>>>>>>>>>> Sebastian
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> On Tue, Sep 29, 2026 at 2:03 PM Divjot Arora via dev <
>>>>>>>>>>> dev@parquet.apache.org>
>>>>>>>>>>> wrote:
>>>>>>>>>>> 
>>>>>>>>>>>> I agree with moving forward with the proposal and
>>> renaming
>>>>>> VECTOR
>>>>>>>> to
>>>>>>>>>>>> FINITE_VECTOR or something
>>>>>>>>>>>> similar if others feel that better communicates its
>>>> intent. I
>>>>>> see
>>>>>>>>> value
>>>>>>>>>>> in
>>>>>>>>>>>> having a vector type that guarantees
>>>>>>>>>>>> its values are mathematically-correct vectors for a few
>>>>>> reasons:
>>>>>>>>>>>> 
>>>>>>>>>>>> 1. As others have mentioned, this is similar to the
>>> STRING
>>>>>>> logical
>>>>>>>>> type
>>>>>>>>>>>> mandating UTF-8.
>>>>>>>>>>>> 2. It is not possible to rely on statistics for
>> detecting
>>>> NaN
>>>>>> or
>>>>>>>>> +-inf
>>>>>>>>>>>> values as all statistics are optional.
>>>>>>>>>>>> To my knowledge, there is no precedent for requiring
>> that
>>>>>> writers
>>>>>>>>> emit
>>>>>>>>>>>> stats for specific logical types.
>>>>>>>>>>>> 3. Without statistics, NaN/inf values could flow
>> through
>>>> the
>>>>>>> engine
>>>>>>>>> and
>>>>>>>>>>>> block them for specific functionality.
>>>>>>>>>>>> This seems to be Lance's behavior. However, I don't
>> think
>>>>> this
>>>>>> is
>>>>>>>> as
>>>>>>>>>> easy
>>>>>>>>>>>> as it sounds and Lance has run
>>>>>>>>>>>> into several bugs due to this ([1] [2] for examples).
>>> Their
>>>>>>> Python
>>>>>>>>> [3]
>>>>>>>>>>> and
>>>>>>>>>>>> Rust [4] APIs default to erroring on
>>>>>>>>>>>> vectors that contain NaN values.
>>>>>>>>>>>> 
>>>>>>>>>>>> I agree that Parquet is a general-purpose format, but I
>>>> don't
>>>>>>> think
>>>>>>>>>> this
>>>>>>>>>>>> implies the lack of domain-specific types.
>>>>>>>>>>>> Instead, this broadens the use cases where Parquet
>> would
>>> be
>>>>>>>>> considered
>>>>>>>>>> a
>>>>>>>>>>>> good fit. I also fully agree with
>>>>>>>>>>>> continuing work on a general fixed-size list type if we
>>>>> believe
>>>>>>>> there
>>>>>>>>>> are
>>>>>>>>>>>> non-vector use cases that would benefit
>>>>>>>>>>>> from such guarantees at the storage layer.
>>>>>>>>>>>> 
>>>>>>>>>>>> [1] https://github.com/lance-format/lance/pull/3648
>>>>>>>>>>>> [2] https://github.com/lance-format/lance/issues/8607
>>> (see
>>>>>>> point 2
>>>>>>>>> in
>>>>>>>>>>>> "root
>>>>>>>>>>>> cause")
>>>>>>>>>>>> [3]
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>> 
>> https://github.com/lancedb/lancedb/blob/0327618a62483feb6dca2336549845e5887cd48f/python/python/lancedb/table.py#L4807-L4911
>>>>>>>>>>>> [4]
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>> 
>> https://github.com/lancedb/lancedb/blob/0327618a62483feb6dca2336549845e5887cd48f/rust/lancedb/src/table/add_data.rs#L105-L110
>>>>>>>>>>>> 
>>>>>>>>>>>> On Mon, Sep 28, 2026 at 6:19 PM Philipp Fischbeck via
>>> dev <
>>>>>>>>>>>> dev@parquet.apache.org> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>>> I also favor moving forward with the proposal as is.
>> I
>>>> see
>>>>>>> strong
>>>>>>>>>> value
>>>>>>>>>>>> and
>>>>>>>>>>>>> use cases for the numeric vector type, and I agree
>> with
>>>> the
>>>>>>>>>>> restrictions
>>>>>>>>>>>>> around NaN/Inf values and element types in general.
>>>>> Precedent
>>>>>>>>> exists
>>>>>>>>>> in
>>>>>>>>>>>>> other types that restrict which physical type values
>>> are
>>>>>>>> logically
>>>>>>>>>>> valid
>>>>>>>>>>>>> (e.g., the geospatial types needing to be
>> WKB-encoded).
>>>> A
>>>>>>>> general
>>>>>>>>>>>>> fixed-size list also has value, but I think it makes
>>>> sense
>>>>> to
>>>>>>>>>> separate
>>>>>>>>>>>> the
>>>>>>>>>>>>> two types. The semantics are relevant, and I can see
>>> how
>>>>> the
>>>>>>> two
>>>>>>>>>> types
>>>>>>>>>>>>> might diverge further in the future regarding
>>> statistics
>>>> or
>>>>>>>>>> encodings.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> I would be okay with renaming the vector type, but it
>>>> seems
>>>>>>> like
>>>>>>>>> the
>>>>>>>>>>>> VECTOR
>>>>>>>>>>>>> name is already used across other engines mentioned
>> in
>>>> this
>>>>>>>> thread.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Best,
>>>>>>>>>>>>> Philipp
>>>>>>>>>>>>> 
>>>>>>>>>>>>> On Sat, Sep 26, 2026 at 4:37 PM Gang Wu <
>>>> ustcwg@gmail.com>
>>>>>>>> wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> To separate orthogonal concerns, I feel we should
>>> treat
>>>>> the
>>>>>>>>> general
>>>>>>>>>>>>>> FixedSizeList as the underlying physical layout,
>> and
>>>>>> separate
>>>>>>>> it
>>>>>>>>>> from
>>>>>>>>>>>> the
>>>>>>>>>>>>>> discussion of a specialized VECTOR (or broader ML
>>>> tensor)
>>>>>>>> logical
>>>>>>>>>>> type.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Currently, the immediate need is to define a
>>>> specialized
>>>>>>> VECTOR
>>>>>>>>>> type
>>>>>>>>>>> so
>>>>>>>>>>>>>> that different engines and formats have a common
>>>> semantic
>>>>>>>>> contract
>>>>>>>>>>> for
>>>>>>>>>>>>>> interoperability.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> If the main point of contention is just the
>>>> NaN/Inf/null
>>>>>>>>>> limitations,
>>>>>>>>>>>>> maybe
>>>>>>>>>>>>>> we can resolve this by adding a qualification to
>> the
>>>> name
>>>>>>> (like
>>>>>>>>>>> Andrew
>>>>>>>>>>>>>> suggested, e.g., VECTOR_EMBEDDING or FINITE_VECTOR)
>>> to
>>>>>> avoid
>>>>>>>>>>> confusion
>>>>>>>>>>>>> with
>>>>>>>>>>>>>> broader ML use cases. Alternatively, we could
>>>> introduce a
>>>>>>>> boolean
>>>>>>>>>>> flag
>>>>>>>>>>>>>> (e.g., optional bool VectorType.finite) as proposed
>>>>>> earlier.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On Fri, Sep 25, 2026 at 3:52 AM Andrew Lamb <
>>>>>>>>>> andrewlamb11@gmail.com>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> I am a little confused about the points of
>>>> disagreement
>>>>>>> here.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Is it that the name `VECTOR` is too general for
>> the
>>>>>>> specific
>>>>>>>>> use
>>>>>>>>>>> case
>>>>>>>>>>>>> of
>>>>>>>>>>>>>>> fixed length arrays with non-finite values used
>> for
>>>>>> vector
>>>>>>>>>>>> embeddings?
>>>>>>>>>>>>>> If
>>>>>>>>>>>>>>> so, maybe we could add a qualification to the
>> name
>>>>>>>>>> (FINITE_VECTOR?
>>>>>>>>>>>>>>> FEATURE_VECTOR? VECTOR_EMBEDDING?)
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> It seems like there is precedent for defining
>>> logical
>>>>>> types
>>>>>>>>> that
>>>>>>>>>>>>> specify
>>>>>>>>>>>>>>> certain physical types and restrict their
>> contents
>>> to
>>>>>> help
>>>>>>>>>> readers
>>>>>>>>>>>>>>> interpret meaning, for example string[1] or JSON;
>>>>>> Defining
>>>>>>>> one
>>>>>>>>>> for
>>>>>>>>>>>>>> vector
>>>>>>>>>>>>>>> embeddings seems reasonable enough to me.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Andrew
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> [1]:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>> 
>> https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#string
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> On Thu, Sep 24, 2026 at 3:34 PM Daniel Weeks <
>>>>>>>>> dweeks@apache.org>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Well, Parquet is not a single-purpose format.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Vector stores can choose whatever policy fits
>>>> their
>>>>>>>> usage,
>>>>>>>>>> but
>>>>>>>>>>>>> we're
>>>>>>>>>>>>>>>>> talking about a general-purpose file format
>>> that
>>>>> aims
>>>>>>> to
>>>>>>>> be
>>>>>>>>>>>> broadly
>>>>>>>>>>>>>>>>> applicable.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> I agree that the format is intended to be
>> general
>>>>>>> purpose,
>>>>>>>>> but
>>>>>>>>>>>> types
>>>>>>>>>>>>>>> within
>>>>>>>>>>>>>>>> the format define specific representation and
>>>>> semantics
>>>>>>> to
>>>>>>>>>>> achieve
>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>> goal.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> "Vector" has a more general sense than ML and
>>>> it's
>>>>>>> quite
>>>>>>>>>>> trivial
>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>> produce vectors with non-finite values using
>>> e.g.
>>>>>> NumPy
>>>>>>>> or
>>>>>>>>>>> Pandas
>>>>>>>>>>>>> (or
>>>>>>>>>>>>>>>>> any other piece of dataframe or scientific
>>>>> computing
>>>>>>>>>> software).
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> The justification for restricting non-finite
>>>> values
>>>>> is
>>>>>>>>>> directly
>>>>>>>>>>>>>> related
>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>> the semantic meaning of a vector.  Vectors have
>>>>>> specific
>>>>>>>>>>>>>> characteristics
>>>>>>>>>>>>>>>> (dimension, magnitude, direction, etc).
>>> Non-finite
>>>>>>> values
>>>>>>>>> are
>>>>>>>>>>>>> invalid
>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>> break those characteristics.  I'd liken it to
>>>> storing
>>>>>>>>> non-utf8
>>>>>>>>>>>> values
>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>> string and still calling it a string.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> That also differentiates a vector from a fixed
>>>> length
>>>>>>> list,
>>>>>>>>>> which
>>>>>>>>>>>> is
>>>>>>>>>>>>>> more
>>>>>>>>>>>>>>>> general and does not carry the same semantic
>>>> meaning.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> I also don't think we need to bundle all
>>> similarly
>>>>>> shaped
>>>>>>>> use
>>>>>>>>>>> cases
>>>>>>>>>>>>>> into
>>>>>>>>>>>>>>>> the same solution (e.g. flxed size list).
>> Vector
>>>> is
>>>>> a
>>>>>>> well
>>>>>>>>>>>>> established
>>>>>>>>>>>>>>>> data type at this point and aligns with the
>>> general
>>>>>>> purpose
>>>>>>>>>>> nature
>>>>>>>>>>>> of
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> format (many other formats and databases
>> natively
>>>>>> support
>>>>>>>>>> this).
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> -Dan
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> On Thu, Sep 24, 2026 at 11:55 AM Antoine
>> Pitrou <
>>>>>>>>>>>> antoine@python.org>
>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Le 24/09/2026 à 20:11, Russell Spitzer a
>> écrit
>>> :
>>>>>>>>>>>>>>>>>> I think something I haven't seen yet is the
>>>> clear
>>>>>>>>> use-case
>>>>>>>>>>> for
>>>>>>>>>>>>>>>> non-finite
>>>>>>>>>>>>>>>>>> values within a vector type. There has
>> been a
>>>> lot
>>>>>> of
>>>>>>>>>>>> theoretical
>>>>>>>>>>>>>>>>> discussion
>>>>>>>>>>>>>>>>>> here about use cases but what toolings
>>>> currently
>>>>>>>> produce
>>>>>>>>>>> these
>>>>>>>>>>>>>>> vectors
>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>> how do they function in downstream engines?
>>> I'd
>>>>>> like
>>>>>>> to
>>>>>>>>>> limit
>>>>>>>>>>>>>>>> introducing
>>>>>>>>>>>>>>>>>> types that have a wide range of values
>> which
>>>> are
>>>>>>> simply
>>>>>>>>> are
>>>>>>>>>>>>> failure
>>>>>>>>>>>>>>>> modes
>>>>>>>>>>>>>>>>>> of different flavors
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> "Vector" has a more general sense than ML and
>>>> it's
>>>>>>> quite
>>>>>>>>>>> trivial
>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>> produce vectors with non-finite values using
>>> e.g.
>>>>>> NumPy
>>>>>>>> or
>>>>>>>>>>> Pandas
>>>>>>>>>>>>> (or
>>>>>>>>>>>>>>>>> any other piece of dataframe or scientific
>>>>> computing
>>>>>>>>>> software).
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> It's not like the general concept of a vector
>>> is
>>>> a
>>>>>> IETF
>>>>>>>>>>> standard
>>>>>>>>>>>> or
>>>>>>>>>>>>>>>>> anything, so analogies with JSON or UTF8
>> (where
>>>> the
>>>>>>>> values
>>>>>>>>>> are
>>>>>>>>>>>>>> expected
>>>>>>>>>>>>>>>>> to be valid according to the respective
>> specs)
>>> do
>>>>> not
>>>>>>>>> apply.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> This discussion would IMHO have a better
>> chance
>>>> of
>>>>>>>>> converging
>>>>>>>>>>> if
>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>> separated the two orthogonal concerns of:
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 1. representing fixed-length list/vector data
>>> in
>>>>>>> Parquet
>>>>>>>>>> (with
>>>>>>>>>>> a
>>>>>>>>>>>>>>>>> potential limitation to non-nested types)
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 2. representing the information that a given
>>>> column
>>>>>> (or
>>>>>>>>>> column
>>>>>>>>>>>>> chunk,
>>>>>>>>>>>>>>>>> etc.) does not contain non-finite values
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> ... both of which are useful on their own
>>> right,
>>>>> but
>>>>>> do
>>>>>>>> not
>>>>>>>>>>> have
>>>>>>>>>>>> to
>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>> conflated and bundled together.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Regards
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Antoine.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>> 
>> 

Re: [DISCUSS] Numeric Vector type: physical representation (outcome of focused call)

[Andrew Bell <an...@gmail.com>]

On Wed, Oct 14, 2026, 3:45 PM Daniel Weeks <dw...@apache.org> wrote:

In parquet, establishing semantic meaning is the purpose of logical types.
> The logical type separates how to interpret a value from its physical
> representation.  We can represent a timestamp with INT64 or even a string
> value, so why have a type for it? Even though a FIXED_SIZED_LIST and VECTOR
> may have the exact same representation and performance, they have different
> semantic meaning.


I'll take this one step further... Why is there any debate about this at
all? Why can't a type simply be added to the *specification* as an
administrative detail and be done? Why isn't it trivial to add logical
types? No change needs to be made to any existing software (or at least
adding logical types shouldn't necessitate changes).

--
Andrew Bell
andrew.bell.ia@gmail.com

Re: [DISCUSS] Numeric Vector type: physical representation (outcome of focused call)

[Daniel Weeks <dw...@apache.org>]

Hey Jan,

There's a lot that I disagree with respect to the framing of these
arguments and how we should be thinking about introducing vector as a
logical type.  There's a repeated tendency to let the physical
representation or performance drive discussion around types as opposed to
what it represents.

In parquet, establishing semantic meaning is the purpose of logical types.
The logical type separates how to interpret a value from its physical
representation.  We can represent a timestamp with INT64 or even a string
value, so why have a type for it? Even though a FIXED_SIZED_LIST and VECTOR
may have the exact same representation and performance, they have different
semantic meaning.  Databases and engines across the ecosystem already
recognize this differentiation and have separate representations.  The
choice to separate these isn't because the discussion is too tedious;
they're fundamentally different things.

The finite requirement is what makes a vector a reflection of what it
represents.  As we've discussed in the thread, non-finite values in the
representation violate the properties of a vector and the value ceases to
be a vector.  On the other hand, a 0 vector is still a vector (you can
argue the utility of it, but it is a valid vector).  Implementations can
choose what they validate or how to handle execution errors; that's an
implementation concern, not a requirement.  Operations performed on a
vector depend on the context in which they are used, but the representation
doesn't need to solve scenarios where that application doesn't make sense.

The guarantees or context needed to use a vector do not need to be inherent
in the type for it to be a useful type.  We don't need the currency used
for decimal or whether a floating point is a normalized value as part of
the type to make use of those types.  This conflates the use cases with the
type system and unnecessarily attempts to draw the context into the type
definition.  Similarly, you don't need to capture normalization, model,
etc. as part of the type for it to be useful.

So the reason we want to add a vector type (or FINITE_VECTOR) is because we
want to store vectors with the semantic meaning and requirements based on
what they represent.

-Dan


On Wed, Oct 14, 2026 at 1:22 AM Jan Finis <jp...@gmail.com> wrote:

> A bit late to the party, but here we go:
>
> I have the strong sentiment that the FINITE_VECTOR type as it is currently
> being discussed will not be very useful and just increase implementation
> complexity. The arguments for the type are all quite weak (details see
> below). The FINITE part doesn't seem to yield any value and is not a useful
> restriction by itself. The vector (i.e. fixed length) part should rather be
> generalized instead of making a one-off thing. In general, the existing
> types are already working well for uncompressed vectors. The real challenge
> that would bring real benefits is supporting quantization. This is where
> you get orders of magnitude improvements to storage space and processing
> speed.
>
> Let's first discuss the FINITE part. So far, the only arguments I have read
> for this are:
>
> 1. many other vector databases have it
> 2. vectors are the hot stuff right now, so Parquet better boast having
> "native" type support for vectors
> 3. by focussing on this common special case, we can make rapid progress,
> while a general solution would take too long to specify
> 4. to give a guarantee to readers that no INF/NaN/NULL will be in the
> vector
>
> the first two arguments are quite weak by themselves. Vector databases are
> very specialized engines. Such engines can of course be very opinionated on
> their data types. A general format like Parquet must instead weigh whether
> a special type is worth the added complexity over a more general, widely
> applicable type. In this case, it is IMHO not at all. And just having a
> type called "vector" for advertisement reasons to make Parquet appear more
> modern seems a non-goal in the first place.
>
> The third "time to market" argument is valid for systems that don't need a
> stable interface. It is a very dangerous attitude for formats like Parquet
> that must be stable long term and should have a consolidated, concise,
> non-redundant type system. For such long term storage formats, we need to
> take the time to build the right thing, as any shortcuts we take will stay
> as blemishes in the format forever. If I read the discussion correctly,
> this is actually the main argument as of now: We have found that it is hard
> to align on how a fixed size list type should look like, so we want to go
> the easier route of a specialized type. This seems like a
> dangerous shortcut fueled by feature recency bias. Ask yourself: Would we
> want this, if Parquet already had a fixed size list type?
>
> So finally, the only technical argument is the fourth one: "It signals
> semantics and ensures that no bad data lands in the format". I see two big
> problems with this:
>
> 1. Readers *still* need to test for NaN / INF because Parquet files may be
> corrupt. Nothing can stop a malicious or buggy writer from physically
> writing NaN or INF. *So you save no single line of code in a reader. You
> save no branches, as you still need to handle the INF/NaN case.* So this
> doesn't help with performance at all. The only difference is, that the
> error message changes from "found an invalid INF value in the data" to
> "Your parquet file seems to be corrupt". It is actually quite the opposite,
> readers and writers get more complex having to handle this new type.
>
> If you now argue that it's not about performance, it's about defined
> semantics and erroring out early in writers, then the second problem
> arises:
>
> 2. The proposed no-NaN/INF/NULL guarantee is just one of multiple
> guarantees that you need for sensible vector processing. If "no
> INF/NaN/NULL" was the only guarantee you needed, I would agree that
> FINITE_VECTOR is useful, but it is by no means the only guarantee you need,
> instead, depending on the distance metric, you also need the following
> guarantees. E.g.:
>
>    - No vector may be the 0 vector, otherwise, e.g., cosine similarity is
>    undefined. Cosine similarity is often used as default, so this is
> actually
>    quite important in practice.
>    - If we want to use a dot product, then all vectors need to guarantee a
>    length of 1. This would actually be quite useful for performance
> reasons.
>    - The vectors must come from the same model as the vector used for the
>    search. Otherwise, the search yields a numeric result, but that result
> has
>    no sensible meaning.
>
> So, what use is a type that just guarantees part of the things you need for
> vector search but not others? You still need additional guarantees in
> addition to that type. What makes INF/NaN so important in comparison to,
> say, "no 0 vector present", that it warrants an own type, while the other
> guarantees do not? If we really want this type as a basis for cross-engine
> sharing of vectors, then it needs to give *all* guarantees.
>
> I'm not against a specialized vector type if it really carries *all*
> information to be useful by itself. If it carried all that information,
> then a reading engine could indeed validate that only meaningful vector
> searches are executed. But as long as it is just a half-baked thing, it is
> not useful by itself and just a burden on each implementer and maintainer.
> If we need extra information anyway, why not just stick to a general type
> like VECTOR (without FINITE).
>
> If you're a proponent of FINITE_VECTOR, ask yourself (or tell us) what
> advantage you will *really* get over just a (potentially fixed size) List
> of Float? As discussed, it's not performance. It's also not a semantic
> marker that is sufficient by itself. It's just a small hint that allows
> writers to error out once they see a NaN or INF in the input, while they
> will still happily write a bogus 0 vector. This seems to be of zero
> benefit.
>
> Anecdotally, we currently handle petabytes of vectors in production in
> Data360. We simply store them in a List of Float, with an optimized R-Level
> reader implementation that rapidly checks that all vectors are the same
> length. This solution works just fine. The overhead of R-level decoding is
> barely measurable. The check is super fast and R-levels are tiny in
> comparison to the payload vectors, so also the added I/O is negligible.
> This is also why I'm so firm in my sentiment: We are running a production
> grade system processing trillions of vectors right now and see no big
> issues with the types that Parquet currently provides.
>
> As I don't just want to be a naysayer and point out problems with the
> current design, here are the directions that I would propose indeed:
>
>    - a general fixed size list type. Don't do shortcuts just because the
>    discussion is too tedious.
>    - a vector type that indeed carries enough information to be really
>    useful for vector search by itself. (e.g., model name / version,
> optional
>    guarantees for no 0 vector / normalized vector length). With such a
> type,
>    you could really do well defined vector searches even if the writing
> engine
>    is completely different to the reading one.
>    - Standardize quantizations. This will be a real game changer for
>    performance.
>
> Cheers,
> Jan
>
>
>
>
>
>
> Am Mi., 14. Okt. 2026 um 03:36 Uhr schrieb Gang Wu <us...@gmail.com>:
>
> > I think separating them into FINITE_VECTOR and FIXED_LENGTH_LIST is a
> good
> > idea, though I think renaming FINITE_VECTOR to VECTOR looks more natural.
> > Should we move forward with proposing FINITE_VECTOR/VECTOR first?
> >
> > On Tue, Oct 13, 2026 at 6:26 PM Alkis Evlogimenos via dev <
> > dev@parquet.apache.org> wrote:
> >
> > > I suggest we go with VECTOR for the finite vector to keep the name the
> > same
> > > as other engines/type systems. If we want a non-finite one in the
> future
> > we
> > > can name that something else or we add fixed size list that will serve
> > the
> > > same purpose without a special name.
> > >
> > >
> > > On Thu, Oct 8, 2026 at 10:20 PM Ryan Blue <bl...@apache.org> wrote:
> > >
> > > > A logical type would:
> > > > - Describe data for users and readers to understand what the file
> > > contains
> > > > - Signal that repetition and definition levels will reflect a fixed
> > > number
> > > > of elements, which implementations can use to optimize decoding
> > > > - Guarantee that there are no null elements (because the type cannot
> be
> > > > applied to a list with optional elements)
> > > > - [FINITE_VECTOR] Guarantee that no elements are NaN or Infinity
> values
> > > so
> > > > that the vectors have a finite length and are suitable for distance
> > > > calculations
> > > >
> > > > The finite value restriction is a restriction on writers to prevent
> bad
> > > > data from being written.
> > > >
> > > > Ignoring logical types that are not understood is a rule for
> readers. A
> > > > reader can handle vectors and pass it on to another layer. For
> > instance,
> > > a
> > > > Parquet reader that doesn't support geometry passes values as
> `binary`.
> > > > That binary value can be passed to a geometry UDF, even if the
> > underlying
> > > > engine and Parquet reader doesn't have support.
> > > >
> > > > A tool that doesn't understand FINITE_VECTOR can similarly pass a
> list
> > of
> > > > floats along to a UDF. But it is still important to the user that bad
> > > > values are not present in the data. In your example, bad values are
> > > vectors
> > > > that cause the centroid calculation to fail, like vectors with
> infinite
> > > > length.
> > > >
> > > > Ryan
> > > >
> > > > On Thu, Oct 8, 2026 at 12:54 PM Will Edwards via dev <
> > > > dev@parquet.apache.org>
> > > > wrote:
> > > >
> > > > > Could we clarify the use cases for a VECTOR annotation on
> > LIST<FLOAT>?
> > > > > Specifically, what does it make possible that's impossible today,
> and
> > > > what
> > > > > does it make easier that's currently hard?
> > > > >
> > > > > I'm struggling to find a strong answer. I'd like every engine to
> > offer
> > > > > vector functions on lists of floats without needing a new logical
> > type.
> > > > >
> > > > > To be clear, I'm still enthusiastic about a vector type that
> > describes
> > > > how
> > > > > to unpack and scale composite representations (quantised codes plus
> > > > scales,
> > > > > codebooks, etc.), in the direction ML is going. That adds
> information
> > > > > engines don't have. A label on LIST<FLOAT> doesn't: engines already
> > > > > understand the physical type, and many already offer vector
> functions
> > > on
> > > > > it. I don't see a performance benefit, and NaN/null/Inf checks are
> > > cheap
> > > > > enough that engines should do them anyway. The label also doesn't
> say
> > > > what
> > > > > you can do with the column. Knowing that a column holds embeddings
> > > > doesn't
> > > > > say how they can be used, just as zipcode and age are both integers
> > but
> > > > > multiplying them is meaningless.
> > > > >
> > > > > My concern is the opposite effect. We're establishing that engines
> > > should
> > > > > ignore logical types they don't understand. So a tool that doesn't
> > know
> > > > > VECTOR will read the column as LIST<FLOAT>, let the user compute
> > > > centroids
> > > > > (say), and write plain LIST<FLOAT> back out. If other engines
> > restrict
> > > > > embedding functions to VECTOR-annotated columns, that file stops
> > > working
> > > > > with them, through no fault of the user. An annotation that doesn't
> > > > change
> > > > > what the data is shouldn't be able to make it unusable.
> > > > >
> > > > > Am I missing a use case?
> > > > >
> > > > > Best, Will
> > > > >
> > > > > On Thu, 8 Oct 2026 at 19:53, Steven Wu <st...@gmail.com>
> wrote:
> > > > >
> > > > > > I also think a FINITE_VECTOR type brings a lot of values to many
> > use
> > > > > cases.
> > > > > > It is would be great to make progress on the FINITE_VECTOR, and
> not
> > > to
> > > > > fit
> > > > > > every use cases into the FIXED_LENGTH_LIST.
> > > > > >
> > > > > > On Mon, Oct 5, 2026 at 1:26 PM Ryan Blue <bl...@apache.org>
> wrote:
> > > > > >
> > > > > > > Going through this thread again, it looks like there are a few
> > > things
> > > > > > that
> > > > > > > people generally agree on:
> > > > > > >
> > > > > > > - In the short term, we want to introduce a logical type or
> > types;
> > > a
> > > > > > better
> > > > > > > physical representation can be done later. We can focus on
> > logical
> > > > > types.
> > > > > > > - There is precedent in logical types for restricting values,
> > like
> > > > UTF8
> > > > > > > strings to contain only UTF-8 codepoints. I think we also agree
> > > that
> > > > we
> > > > > > > could restrict vector elements to non-nested or primitive
> types.
> > > > > > > - There are many examples of databases that restrict their
> vector
> > > > types
> > > > > > to
> > > > > > > non-null, finite elements.
> > > > > > > - Some people want to have write-time guarantees that vectors
> > have
> > > > only
> > > > > > > non-null, finite elements
> > > > > > > - Some people want to extend this work to include tensors,
> > shorter
> > > > > > floating
> > > > > > > point representations, variable lengths, NaN or Infinity
> > elements,
> > > > etc.
> > > > > > >
> > > > > > > It sounds like we are nearly there, but have two halves of the
> > > > > community
> > > > > > > that care about different things. That just means we should
> have
> > > > > multiple
> > > > > > > types. It's entirely reasonable to want a type that allows NaN
> > > > values,
> > > > > > and
> > > > > > > it's also reasonable for those of us building databases, table
> > > > > standards,
> > > > > > > and vector indexes to care about guarantees that elements are
> > > finite.
> > > > > > > There's more than enough examples to show that this is a
> > legitimate
> > > > use
> > > > > > > case.
> > > > > > >
> > > > > > > Since I don't see the concerns of the different use cases
> > changing,
> > > > it
> > > > > > > looks like we need to find a compromise to move forward. I
> > propose
> > > > that
> > > > > > we
> > > > > > > introduce a FINITE_VECTOR type and separately a
> FIXED_LENGTH_LIST
> > > or
> > > > > > > another VECTOR type that allows infinite and NaN values.
> > > > > > >
> > > > > > > Can we agree on a compromise? Is there a strong reason to try
> to
> > > fit
> > > > > > these
> > > > > > > use cases into a single type? What if that means not making
> > > progress?
> > > > > > >
> > > > > > > Ryan
> > > > > > >
> > > > > > > On Mon, Oct 5, 2026 at 12:15 PM Philipp Fischbeck via dev <
> > > > > > > dev@parquet.apache.org> wrote:
> > > > > > >
> > > > > > > > Thinking about this more, I am also fine with pulling out the
> > > > > “NaN/Inf”
> > > > > > > > requirement. Either into a type boolean (my preference), or
> > > putting
> > > > > no
> > > > > > > > restrictions on the Parquet level, allowing all IEEE 754
> float
> > > > > numbers.
> > > > > > > If
> > > > > > > > some top-level application wants to put additional
> constraints
> > on
> > > > the
> > > > > > > data
> > > > > > > > type, like Iceberg not allowing NaN/infinity, it can still do
> > so
> > > > one
> > > > > > > layer
> > > > > > > > above. It seems like this was the biggest point of discussion
> > > > > > remaining.
> > > > > > > > All these options work for me, and we should move forward
> with
> > > this
> > > > > > > > proposal.
> > > > > > > >
> > > > > > > > Concerning additional element types like bf16, they can be
> > > > introduced
> > > > > > as
> > > > > > > > Parquet types (at least in the current LIST-based spec
> > proposal).
> > > > > > > >
> > > > > > > > Concerning Will’s suggestion around representing VECTOR as a
> > > > variable
> > > > > > > > length list with a MIN/MAX_LENGTH metadata restriction: I
> think
> > > > that
> > > > > a
> > > > > > > > dedicated logical VECTOR type still provides enough benefits
> by
> > > > > > > describing
> > > > > > > > the semantics of the data, and allowing to capture future
> > changes
> > > > > > around
> > > > > > > > statistics. In that case, the length is better captured as an
> > > exact
> > > > > > > length
> > > > > > > > for this type specifically. I’m also not sure about other use
> > > cases
> > > > > for
> > > > > > > > MIN_LENGTH, so introducing a new type constraint system just
> > for
> > > > the
> > > > > > > > MIN_LENGTH=MAX_LENGTH case for vectors doesn’t have enough
> > value
> > > in
> > > > > my
> > > > > > > > view.
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Philipp
> > > > > > > >
> > > > > > > >
> > > > > > > > On Thu, Oct 1, 2026 at 9:00 AM Will Edwards via dev <
> > > > > > > > dev@parquet.apache.org>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > +1 Sebastian, a vector type suggests usage beyond
> normalized
> > > > > > embeddings
> > > > > > > > and
> > > > > > > > > engines really aren't going to be able to elide all the
> > checks
> > > > > > anyway.
> > > > > > > > >
> > > > > > > > > Antoine, you suggested we explore metadata solutions too :)
> > > > > > > > >
> > > > > > > > > Today, a database supporting CHAR(N) or VARCHAR(n) can't
> > > express
> > > > > this
> > > > > > > in
> > > > > > > > > Parquet.  Engines can write opaque blobs they understand in
> > the
> > > > > > > metadata
> > > > > > > > > (e.g. Arrow does) so the extra type info round-trips, but
> > this
> > > > > isn't
> > > > > > > > > standardized, portable, nor interoperable.
> > > > > > > > >
> > > > > > > > > So perhaps standardizing constraint metadata or new Thrift
> > > fields
> > > > > (my
> > > > > > > > pref)
> > > > > > > > > can generically support VECTOR(N) whilst also being more
> > > broadly
> > > > > > useful
> > > > > > > > for
> > > > > > > > > other use cases and data types?
> > > > > > > > >
> > > > > > > > > Imagine metadata or Thrift fields to tag the MIN_LENGTH and
> > > > > > MAX_LENGTH
> > > > > > > > for
> > > > > > > > > Parquet's variable-length physcial types.  These would
> count
> > > > > logical
> > > > > > > > units
> > > > > > > > > (e.g. UTF8 code points for STRING) rather than bytes.
> > > > > > > > >
> > > > > > > > > Then one engine's VECTOR(N) can be written to Parquet as
> > > LIST(N)
> > > > > with
> > > > > > > > > MIN_LENGTH = MAX_LENGTH = N and then readers - even
> different
> > > > > > engines -
> > > > > > > > can
> > > > > > > > > pick up these constraints if they support doing so.  We
> don't
> > > > > > actually
> > > > > > > > need
> > > > > > > > > a logical type to express a VECTOR then.
> > > > > > > > >
> > > > > > > > > This also allows standard SQL's CHAR(N) and VARCHAR(N) to
> be
> > > > > > expressed
> > > > > > > on
> > > > > > > > > UTF8 columns etc.
> > > > > > > > >
> > > > > > > > > This also seems generally useful in the future as we
> approach
> > > > > > > expressing
> > > > > > > > > other tensor data shapes and types.
> > > > > > > > >
> > > > > > > > > This won't change the mechanics: the storage type doesn't
> > > enforce
> > > > > > these
> > > > > > > > > constraints and engines must check them. It's previously
> > > > > established
> > > > > > > that
> > > > > > > > > performance is just a software problem and there is no
> > > > fundamental
> > > > > > > > > performance advantage to the fixed size types in
> Dremel-like
> > > > > formats
> > > > > > > like
> > > > > > > > > Parquet.
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > > Will
> > > > > > > > >
> > > > > > > > > On Wed, 30 Sept 2026 at 16:08, Sebastian Baunsgaard <
> > > > > > > > baunsgaard@apache.org
> > > > > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > After listening in on the last meeting Wednesday, I have
> > been
> > > > > > > thinking
> > > > > > > > > > about this quite a bit, and thought I would share my 2
> > cents.
> > > > > > > > > >
> > > > > > > > > > I suggest not requiring VECTOR elements to be finite, as
> > > > proposed
> > > > > > in
> > > > > > > > the
> > > > > > > > > PR
> > > > > > > > > > <https://github.com/apache/parquet-format/pull/624>.
> > > > > > > > > >
> > > > > > > > > > While Divjot and others note the specialization for
> UTF-8,
> > > > > Parquet
> > > > > > > > > already
> > > > > > > > > > defines DOUBLE and FLOAT as IEEE encodings
> > > > > > > > > > <
> > > > > > >
> > > https://github.com/apache/parquet-format/blob/master/README.md#types
> > > > > > > > >.
> > > > > > > > > >
> > > > > > > > > > Making NaN or +/-Inf illegal inside VECTOR would be a new
> > > > > > restriction
> > > > > > > > on
> > > > > > > > > > those same underlying physical types. Scalar FLOAT/DOUBLE
> > > > columns
> > > > > > > have
> > > > > > > > > > never had such a rule.
> > > > > > > > > >
> > > > > > > > > > It would also not match how the systems that actually
> > produce
> > > > > these
> > > > > > > > > vectors
> > > > > > > > > > treat floats. Most of them use IEEE floats, e.g. Spark,
> > > DuckDB,
> > > > > > > NumPy,
> > > > > > > > > > PyTorch, TensorFlow, or lower level CUDA, oneMKL, and
> > > OpenBLAS.
> > > > > > > > > >
> > > > > > > > > > pgvector's "all elements must be finite" rule is an
> insert
> > > > > policy.
> > > > > > > > While
> > > > > > > > > I
> > > > > > > > > > think Parquet as a general-purpose format should allow
> them
> > > out
> > > > > of
> > > > > > > the
> > > > > > > > > box.
> > > > > > > > > > It is a user system decision to disallow writing these
> > values
> > > > or
> > > > > to
> > > > > > > > throw
> > > > > > > > > > errors when encountering them.
> > > > > > > > > >
> > > > > > > > > > A type-level ban also does not keep Inf/NaN out of vector
> > > math.
> > > > > > > Finite
> > > > > > > > > > coordinates still overflow. For instance: magnitude =
> > > > > ||[FLOAT_MAX,
> > > > > > > > > > FLOAT_MAX]|| where every coordinate is finite overflows
> to
> > > > +Inf.
> > > > > > > > PyTorch
> > > > > > > > > > documents the same overflow: tensor([1e20, 1e20]).norm()
> is
> > > inf
> > > > > in
> > > > > > > > fp32.
> > > > > > > > > >
> > > > > > > > > > A generated MAGNITUDE(vec) column would then be Inf while
> > the
> > > > > > VECTOR
> > > > > > > > > itself
> > > > > > > > > > was required to be finite. The ban only forbids storing
> > > > specials
> > > > > in
> > > > > > > the
> > > > > > > > > > coordinates, and does not prevent even simple downstream
> > > > > > computations
> > > > > > > > > from
> > > > > > > > > > producing them.
> > > > > > > > > >
> > > > > > > > > > I suggest keeping VECTOR as the type and add an optional
> > bool
> > > > > > > > > > VectorType.finite (default false). Engines that want
> > > > > pgvector-like
> > > > > > > > > > rejection can require the flag = true. Engines that use
> > > NumPy /
> > > > > > > > PyTorch /
> > > > > > > > > > Spark arrays can keep representable IEEE values. For the
> > same
> > > > > > > reason, I
> > > > > > > > > do
> > > > > > > > > > not favor adding yet another type called FINITE_VECTOR.
> > > > > > > > > >
> > > > > > > > > > Best,
> > > > > > > > > > Sebastian
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Tue, Sep 29, 2026 at 2:03 PM Divjot Arora via dev <
> > > > > > > > > > dev@parquet.apache.org>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > I agree with moving forward with the proposal and
> > renaming
> > > > > VECTOR
> > > > > > > to
> > > > > > > > > > > FINITE_VECTOR or something
> > > > > > > > > > > similar if others feel that better communicates its
> > > intent. I
> > > > > see
> > > > > > > > value
> > > > > > > > > > in
> > > > > > > > > > > having a vector type that guarantees
> > > > > > > > > > > its values are mathematically-correct vectors for a few
> > > > > reasons:
> > > > > > > > > > >
> > > > > > > > > > > 1. As others have mentioned, this is similar to the
> > STRING
> > > > > > logical
> > > > > > > > type
> > > > > > > > > > > mandating UTF-8.
> > > > > > > > > > > 2. It is not possible to rely on statistics for
> detecting
> > > NaN
> > > > > or
> > > > > > > > +-inf
> > > > > > > > > > > values as all statistics are optional.
> > > > > > > > > > > To my knowledge, there is no precedent for requiring
> that
> > > > > writers
> > > > > > > > emit
> > > > > > > > > > > stats for specific logical types.
> > > > > > > > > > > 3. Without statistics, NaN/inf values could flow
> through
> > > the
> > > > > > engine
> > > > > > > > and
> > > > > > > > > > > block them for specific functionality.
> > > > > > > > > > > This seems to be Lance's behavior. However, I don't
> think
> > > > this
> > > > > is
> > > > > > > as
> > > > > > > > > easy
> > > > > > > > > > > as it sounds and Lance has run
> > > > > > > > > > > into several bugs due to this ([1] [2] for examples).
> > Their
> > > > > > Python
> > > > > > > > [3]
> > > > > > > > > > and
> > > > > > > > > > > Rust [4] APIs default to erroring on
> > > > > > > > > > > vectors that contain NaN values.
> > > > > > > > > > >
> > > > > > > > > > > I agree that Parquet is a general-purpose format, but I
> > > don't
> > > > > > think
> > > > > > > > > this
> > > > > > > > > > > implies the lack of domain-specific types.
> > > > > > > > > > > Instead, this broadens the use cases where Parquet
> would
> > be
> > > > > > > > considered
> > > > > > > > > a
> > > > > > > > > > > good fit. I also fully agree with
> > > > > > > > > > > continuing work on a general fixed-size list type if we
> > > > believe
> > > > > > > there
> > > > > > > > > are
> > > > > > > > > > > non-vector use cases that would benefit
> > > > > > > > > > > from such guarantees at the storage layer.
> > > > > > > > > > >
> > > > > > > > > > > [1] https://github.com/lance-format/lance/pull/3648
> > > > > > > > > > > [2] https://github.com/lance-format/lance/issues/8607
> > (see
> > > > > > point 2
> > > > > > > > in
> > > > > > > > > > > "root
> > > > > > > > > > > cause")
> > > > > > > > > > > [3]
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/lancedb/lancedb/blob/0327618a62483feb6dca2336549845e5887cd48f/python/python/lancedb/table.py#L4807-L4911
> > > > > > > > > > > [4]
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/lancedb/lancedb/blob/0327618a62483feb6dca2336549845e5887cd48f/rust/lancedb/src/table/add_data.rs#L105-L110
> > > > > > > > > > >
> > > > > > > > > > > On Mon, Sep 28, 2026 at 6:19 PM Philipp Fischbeck via
> > dev <
> > > > > > > > > > > dev@parquet.apache.org> wrote:
> > > > > > > > > > >
> > > > > > > > > > > > I also favor moving forward with the proposal as is.
> I
> > > see
> > > > > > strong
> > > > > > > > > value
> > > > > > > > > > > and
> > > > > > > > > > > > use cases for the numeric vector type, and I agree
> with
> > > the
> > > > > > > > > > restrictions
> > > > > > > > > > > > around NaN/Inf values and element types in general.
> > > > Precedent
> > > > > > > > exists
> > > > > > > > > in
> > > > > > > > > > > > other types that restrict which physical type values
> > are
> > > > > > > logically
> > > > > > > > > > valid
> > > > > > > > > > > > (e.g., the geospatial types needing to be
> WKB-encoded).
> > > A
> > > > > > > general
> > > > > > > > > > > > fixed-size list also has value, but I think it makes
> > > sense
> > > > to
> > > > > > > > > separate
> > > > > > > > > > > the
> > > > > > > > > > > > two types. The semantics are relevant, and I can see
> > how
> > > > the
> > > > > > two
> > > > > > > > > types
> > > > > > > > > > > > might diverge further in the future regarding
> > statistics
> > > or
> > > > > > > > > encodings.
> > > > > > > > > > > >
> > > > > > > > > > > > I would be okay with renaming the vector type, but it
> > > seems
> > > > > > like
> > > > > > > > the
> > > > > > > > > > > VECTOR
> > > > > > > > > > > > name is already used across other engines mentioned
> in
> > > this
> > > > > > > thread.
> > > > > > > > > > > >
> > > > > > > > > > > > Best,
> > > > > > > > > > > > Philipp
> > > > > > > > > > > >
> > > > > > > > > > > > On Sat, Sep 26, 2026 at 4:37 PM Gang Wu <
> > > ustcwg@gmail.com>
> > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > To separate orthogonal concerns, I feel we should
> > treat
> > > > the
> > > > > > > > general
> > > > > > > > > > > > > FixedSizeList as the underlying physical layout,
> and
> > > > > separate
> > > > > > > it
> > > > > > > > > from
> > > > > > > > > > > the
> > > > > > > > > > > > > discussion of a specialized VECTOR (or broader ML
> > > tensor)
> > > > > > > logical
> > > > > > > > > > type.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Currently, the immediate need is to define a
> > > specialized
> > > > > > VECTOR
> > > > > > > > > type
> > > > > > > > > > so
> > > > > > > > > > > > > that different engines and formats have a common
> > > semantic
> > > > > > > > contract
> > > > > > > > > > for
> > > > > > > > > > > > > interoperability.
> > > > > > > > > > > > >
> > > > > > > > > > > > > If the main point of contention is just the
> > > NaN/Inf/null
> > > > > > > > > limitations,
> > > > > > > > > > > > maybe
> > > > > > > > > > > > > we can resolve this by adding a qualification to
> the
> > > name
> > > > > > (like
> > > > > > > > > > Andrew
> > > > > > > > > > > > > suggested, e.g., VECTOR_EMBEDDING or FINITE_VECTOR)
> > to
> > > > > avoid
> > > > > > > > > > confusion
> > > > > > > > > > > > with
> > > > > > > > > > > > > broader ML use cases. Alternatively, we could
> > > introduce a
> > > > > > > boolean
> > > > > > > > > > flag
> > > > > > > > > > > > > (e.g., optional bool VectorType.finite) as proposed
> > > > > earlier.
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Fri, Sep 25, 2026 at 3:52 AM Andrew Lamb <
> > > > > > > > > andrewlamb11@gmail.com>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > I am a little confused about the points of
> > > disagreement
> > > > > > here.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Is it that the name `VECTOR` is too general for
> the
> > > > > > specific
> > > > > > > > use
> > > > > > > > > > case
> > > > > > > > > > > > of
> > > > > > > > > > > > > > fixed length arrays with non-finite values used
> for
> > > > > vector
> > > > > > > > > > > embeddings?
> > > > > > > > > > > > > If
> > > > > > > > > > > > > > so, maybe we could add a qualification to the
> name
> > > > > > > > > (FINITE_VECTOR?
> > > > > > > > > > > > > > FEATURE_VECTOR? VECTOR_EMBEDDING?)
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > It seems like there is precedent for defining
> > logical
> > > > > types
> > > > > > > > that
> > > > > > > > > > > > specify
> > > > > > > > > > > > > > certain physical types and restrict their
> contents
> > to
> > > > > help
> > > > > > > > > readers
> > > > > > > > > > > > > > interpret meaning, for example string[1] or JSON;
> > > > > Defining
> > > > > > > one
> > > > > > > > > for
> > > > > > > > > > > > > vector
> > > > > > > > > > > > > > embeddings seems reasonable enough to me.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Andrew
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > [1]:
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#string
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Thu, Sep 24, 2026 at 3:34 PM Daniel Weeks <
> > > > > > > > dweeks@apache.org>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Well, Parquet is not a single-purpose format.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Vector stores can choose whatever policy fits
> > > their
> > > > > > > usage,
> > > > > > > > > but
> > > > > > > > > > > > we're
> > > > > > > > > > > > > > > > talking about a general-purpose file format
> > that
> > > > aims
> > > > > > to
> > > > > > > be
> > > > > > > > > > > broadly
> > > > > > > > > > > > > > > > applicable.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I agree that the format is intended to be
> general
> > > > > > purpose,
> > > > > > > > but
> > > > > > > > > > > types
> > > > > > > > > > > > > > within
> > > > > > > > > > > > > > > the format define specific representation and
> > > > semantics
> > > > > > to
> > > > > > > > > > achieve
> > > > > > > > > > > > that
> > > > > > > > > > > > > > > goal.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > "Vector" has a more general sense than ML and
> > > it's
> > > > > > quite
> > > > > > > > > > trivial
> > > > > > > > > > > to
> > > > > > > > > > > > > > > > produce vectors with non-finite values using
> > e.g.
> > > > > NumPy
> > > > > > > or
> > > > > > > > > > Pandas
> > > > > > > > > > > > (or
> > > > > > > > > > > > > > > > any other piece of dataframe or scientific
> > > > computing
> > > > > > > > > software).
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >  The justification for restricting non-finite
> > > values
> > > > is
> > > > > > > > > directly
> > > > > > > > > > > > > related
> > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > the semantic meaning of a vector.  Vectors have
> > > > > specific
> > > > > > > > > > > > > characteristics
> > > > > > > > > > > > > > > (dimension, magnitude, direction, etc).
> > Non-finite
> > > > > > values
> > > > > > > > are
> > > > > > > > > > > > invalid
> > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > break those characteristics.  I'd liken it to
> > > storing
> > > > > > > > non-utf8
> > > > > > > > > > > values
> > > > > > > > > > > > > in
> > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > string and still calling it a string.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > That also differentiates a vector from a fixed
> > > length
> > > > > > list,
> > > > > > > > > which
> > > > > > > > > > > is
> > > > > > > > > > > > > more
> > > > > > > > > > > > > > > general and does not carry the same semantic
> > > meaning.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I also don't think we need to bundle all
> > similarly
> > > > > shaped
> > > > > > > use
> > > > > > > > > > cases
> > > > > > > > > > > > > into
> > > > > > > > > > > > > > > the same solution (e.g. flxed size list).
> Vector
> > > is
> > > > a
> > > > > > well
> > > > > > > > > > > > established
> > > > > > > > > > > > > > > data type at this point and aligns with the
> > general
> > > > > > purpose
> > > > > > > > > > nature
> > > > > > > > > > > of
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > format (many other formats and databases
> natively
> > > > > support
> > > > > > > > > this).
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > -Dan
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Thu, Sep 24, 2026 at 11:55 AM Antoine
> Pitrou <
> > > > > > > > > > > antoine@python.org>
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Le 24/09/2026 à 20:11, Russell Spitzer a
> écrit
> > :
> > > > > > > > > > > > > > > > > I think something I haven't seen yet is the
> > > clear
> > > > > > > > use-case
> > > > > > > > > > for
> > > > > > > > > > > > > > > non-finite
> > > > > > > > > > > > > > > > > values within a vector type. There has
> been a
> > > lot
> > > > > of
> > > > > > > > > > > theoretical
> > > > > > > > > > > > > > > > discussion
> > > > > > > > > > > > > > > > > here about use cases but what toolings
> > > currently
> > > > > > > produce
> > > > > > > > > > these
> > > > > > > > > > > > > > vectors
> > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > how do they function in downstream engines?
> > I'd
> > > > > like
> > > > > > to
> > > > > > > > > limit
> > > > > > > > > > > > > > > introducing
> > > > > > > > > > > > > > > > > types that have a wide range of values
> which
> > > are
> > > > > > simply
> > > > > > > > are
> > > > > > > > > > > > failure
> > > > > > > > > > > > > > > modes
> > > > > > > > > > > > > > > > > of different flavors
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > "Vector" has a more general sense than ML and
> > > it's
> > > > > > quite
> > > > > > > > > > trivial
> > > > > > > > > > > to
> > > > > > > > > > > > > > > > produce vectors with non-finite values using
> > e.g.
> > > > > NumPy
> > > > > > > or
> > > > > > > > > > Pandas
> > > > > > > > > > > > (or
> > > > > > > > > > > > > > > > any other piece of dataframe or scientific
> > > > computing
> > > > > > > > > software).
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > It's not like the general concept of a vector
> > is
> > > a
> > > > > IETF
> > > > > > > > > > standard
> > > > > > > > > > > or
> > > > > > > > > > > > > > > > anything, so analogies with JSON or UTF8
> (where
> > > the
> > > > > > > values
> > > > > > > > > are
> > > > > > > > > > > > > expected
> > > > > > > > > > > > > > > > to be valid according to the respective
> specs)
> > do
> > > > not
> > > > > > > > apply.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > This discussion would IMHO have a better
> chance
> > > of
> > > > > > > > converging
> > > > > > > > > > if
> > > > > > > > > > > we
> > > > > > > > > > > > > > > > separated the two orthogonal concerns of:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 1. representing fixed-length list/vector data
> > in
> > > > > > Parquet
> > > > > > > > > (with
> > > > > > > > > > a
> > > > > > > > > > > > > > > > potential limitation to non-nested types)
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 2. representing the information that a given
> > > column
> > > > > (or
> > > > > > > > > column
> > > > > > > > > > > > chunk,
> > > > > > > > > > > > > > > > etc.) does not contain non-finite values
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > ... both of which are useful on their own
> > right,
> > > > but
> > > > > do
> > > > > > > not
> > > > > > > > > > have
> > > > > > > > > > > to
> > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > conflated and bundled together.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Regards
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Antoine.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] Numeric Vector type: physical representation (outcome of focused call)

[Jan Finis <jp...@gmail.com>]

A bit late to the party, but here we go:

I have the strong sentiment that the FINITE_VECTOR type as it is currently
being discussed will not be very useful and just increase implementation
complexity. The arguments for the type are all quite weak (details see
below). The FINITE part doesn't seem to yield any value and is not a useful
restriction by itself. The vector (i.e. fixed length) part should rather be
generalized instead of making a one-off thing. In general, the existing
types are already working well for uncompressed vectors. The real challenge
that would bring real benefits is supporting quantization. This is where
you get orders of magnitude improvements to storage space and processing
speed.

Let's first discuss the FINITE part. So far, the only arguments I have read
for this are:

1. many other vector databases have it
2. vectors are the hot stuff right now, so Parquet better boast having
"native" type support for vectors
3. by focussing on this common special case, we can make rapid progress,
while a general solution would take too long to specify
4. to give a guarantee to readers that no INF/NaN/NULL will be in the vector

the first two arguments are quite weak by themselves. Vector databases are
very specialized engines. Such engines can of course be very opinionated on
their data types. A general format like Parquet must instead weigh whether
a special type is worth the added complexity over a more general, widely
applicable type. In this case, it is IMHO not at all. And just having a
type called "vector" for advertisement reasons to make Parquet appear more
modern seems a non-goal in the first place.

The third "time to market" argument is valid for systems that don't need a
stable interface. It is a very dangerous attitude for formats like Parquet
that must be stable long term and should have a consolidated, concise,
non-redundant type system. For such long term storage formats, we need to
take the time to build the right thing, as any shortcuts we take will stay
as blemishes in the format forever. If I read the discussion correctly,
this is actually the main argument as of now: We have found that it is hard
to align on how a fixed size list type should look like, so we want to go
the easier route of a specialized type. This seems like a
dangerous shortcut fueled by feature recency bias. Ask yourself: Would we
want this, if Parquet already had a fixed size list type?

So finally, the only technical argument is the fourth one: "It signals
semantics and ensures that no bad data lands in the format". I see two big
problems with this:

1. Readers *still* need to test for NaN / INF because Parquet files may be
corrupt. Nothing can stop a malicious or buggy writer from physically
writing NaN or INF. *So you save no single line of code in a reader. You
save no branches, as you still need to handle the INF/NaN case.* So this
doesn't help with performance at all. The only difference is, that the
error message changes from "found an invalid INF value in the data" to
"Your parquet file seems to be corrupt". It is actually quite the opposite,
readers and writers get more complex having to handle this new type.

If you now argue that it's not about performance, it's about defined
semantics and erroring out early in writers, then the second problem arises:

2. The proposed no-NaN/INF/NULL guarantee is just one of multiple
guarantees that you need for sensible vector processing. If "no
INF/NaN/NULL" was the only guarantee you needed, I would agree that
FINITE_VECTOR is useful, but it is by no means the only guarantee you need,
instead, depending on the distance metric, you also need the following
guarantees. E.g.:

   - No vector may be the 0 vector, otherwise, e.g., cosine similarity is
   undefined. Cosine similarity is often used as default, so this is actually
   quite important in practice.
   - If we want to use a dot product, then all vectors need to guarantee a
   length of 1. This would actually be quite useful for performance reasons.
   - The vectors must come from the same model as the vector used for the
   search. Otherwise, the search yields a numeric result, but that result has
   no sensible meaning.

So, what use is a type that just guarantees part of the things you need for
vector search but not others? You still need additional guarantees in
addition to that type. What makes INF/NaN so important in comparison to,
say, "no 0 vector present", that it warrants an own type, while the other
guarantees do not? If we really want this type as a basis for cross-engine
sharing of vectors, then it needs to give *all* guarantees.

I'm not against a specialized vector type if it really carries *all*
information to be useful by itself. If it carried all that information,
then a reading engine could indeed validate that only meaningful vector
searches are executed. But as long as it is just a half-baked thing, it is
not useful by itself and just a burden on each implementer and maintainer.
If we need extra information anyway, why not just stick to a general type
like VECTOR (without FINITE).

If you're a proponent of FINITE_VECTOR, ask yourself (or tell us) what
advantage you will *really* get over just a (potentially fixed size) List
of Float? As discussed, it's not performance. It's also not a semantic
marker that is sufficient by itself. It's just a small hint that allows
writers to error out once they see a NaN or INF in the input, while they
will still happily write a bogus 0 vector. This seems to be of zero benefit.

Anecdotally, we currently handle petabytes of vectors in production in
Data360. We simply store them in a List of Float, with an optimized R-Level
reader implementation that rapidly checks that all vectors are the same
length. This solution works just fine. The overhead of R-level decoding is
barely measurable. The check is super fast and R-levels are tiny in
comparison to the payload vectors, so also the added I/O is negligible.
This is also why I'm so firm in my sentiment: We are running a production
grade system processing trillions of vectors right now and see no big
issues with the types that Parquet currently provides.

As I don't just want to be a naysayer and point out problems with the
current design, here are the directions that I would propose indeed:

   - a general fixed size list type. Don't do shortcuts just because the
   discussion is too tedious.
   - a vector type that indeed carries enough information to be really
   useful for vector search by itself. (e.g., model name / version, optional
   guarantees for no 0 vector / normalized vector length). With such a type,
   you could really do well defined vector searches even if the writing engine
   is completely different to the reading one.
   - Standardize quantizations. This will be a real game changer for
   performance.

Cheers,
Jan






Am Mi., 14. Okt. 2026 um 03:36 Uhr schrieb Gang Wu <us...@gmail.com>:

> I think separating them into FINITE_VECTOR and FIXED_LENGTH_LIST is a good
> idea, though I think renaming FINITE_VECTOR to VECTOR looks more natural.
> Should we move forward with proposing FINITE_VECTOR/VECTOR first?
>
> On Tue, Oct 13, 2026 at 6:26 PM Alkis Evlogimenos via dev <
> dev@parquet.apache.org> wrote:
>
> > I suggest we go with VECTOR for the finite vector to keep the name the
> same
> > as other engines/type systems. If we want a non-finite one in the future
> we
> > can name that something else or we add fixed size list that will serve
> the
> > same purpose without a special name.
> >
> >
> > On Thu, Oct 8, 2026 at 10:20 PM Ryan Blue <bl...@apache.org> wrote:
> >
> > > A logical type would:
> > > - Describe data for users and readers to understand what the file
> > contains
> > > - Signal that repetition and definition levels will reflect a fixed
> > number
> > > of elements, which implementations can use to optimize decoding
> > > - Guarantee that there are no null elements (because the type cannot be
> > > applied to a list with optional elements)
> > > - [FINITE_VECTOR] Guarantee that no elements are NaN or Infinity values
> > so
> > > that the vectors have a finite length and are suitable for distance
> > > calculations
> > >
> > > The finite value restriction is a restriction on writers to prevent bad
> > > data from being written.
> > >
> > > Ignoring logical types that are not understood is a rule for readers. A
> > > reader can handle vectors and pass it on to another layer. For
> instance,
> > a
> > > Parquet reader that doesn't support geometry passes values as `binary`.
> > > That binary value can be passed to a geometry UDF, even if the
> underlying
> > > engine and Parquet reader doesn't have support.
> > >
> > > A tool that doesn't understand FINITE_VECTOR can similarly pass a list
> of
> > > floats along to a UDF. But it is still important to the user that bad
> > > values are not present in the data. In your example, bad values are
> > vectors
> > > that cause the centroid calculation to fail, like vectors with infinite
> > > length.
> > >
> > > Ryan
> > >
> > > On Thu, Oct 8, 2026 at 12:54 PM Will Edwards via dev <
> > > dev@parquet.apache.org>
> > > wrote:
> > >
> > > > Could we clarify the use cases for a VECTOR annotation on
> LIST<FLOAT>?
> > > > Specifically, what does it make possible that's impossible today, and
> > > what
> > > > does it make easier that's currently hard?
> > > >
> > > > I'm struggling to find a strong answer. I'd like every engine to
> offer
> > > > vector functions on lists of floats without needing a new logical
> type.
> > > >
> > > > To be clear, I'm still enthusiastic about a vector type that
> describes
> > > how
> > > > to unpack and scale composite representations (quantised codes plus
> > > scales,
> > > > codebooks, etc.), in the direction ML is going. That adds information
> > > > engines don't have. A label on LIST<FLOAT> doesn't: engines already
> > > > understand the physical type, and many already offer vector functions
> > on
> > > > it. I don't see a performance benefit, and NaN/null/Inf checks are
> > cheap
> > > > enough that engines should do them anyway. The label also doesn't say
> > > what
> > > > you can do with the column. Knowing that a column holds embeddings
> > > doesn't
> > > > say how they can be used, just as zipcode and age are both integers
> but
> > > > multiplying them is meaningless.
> > > >
> > > > My concern is the opposite effect. We're establishing that engines
> > should
> > > > ignore logical types they don't understand. So a tool that doesn't
> know
> > > > VECTOR will read the column as LIST<FLOAT>, let the user compute
> > > centroids
> > > > (say), and write plain LIST<FLOAT> back out. If other engines
> restrict
> > > > embedding functions to VECTOR-annotated columns, that file stops
> > working
> > > > with them, through no fault of the user. An annotation that doesn't
> > > change
> > > > what the data is shouldn't be able to make it unusable.
> > > >
> > > > Am I missing a use case?
> > > >
> > > > Best, Will
> > > >
> > > > On Thu, 8 Oct 2026 at 19:53, Steven Wu <st...@gmail.com> wrote:
> > > >
> > > > > I also think a FINITE_VECTOR type brings a lot of values to many
> use
> > > > cases.
> > > > > It is would be great to make progress on the FINITE_VECTOR, and not
> > to
> > > > fit
> > > > > every use cases into the FIXED_LENGTH_LIST.
> > > > >
> > > > > On Mon, Oct 5, 2026 at 1:26 PM Ryan Blue <bl...@apache.org> wrote:
> > > > >
> > > > > > Going through this thread again, it looks like there are a few
> > things
> > > > > that
> > > > > > people generally agree on:
> > > > > >
> > > > > > - In the short term, we want to introduce a logical type or
> types;
> > a
> > > > > better
> > > > > > physical representation can be done later. We can focus on
> logical
> > > > types.
> > > > > > - There is precedent in logical types for restricting values,
> like
> > > UTF8
> > > > > > strings to contain only UTF-8 codepoints. I think we also agree
> > that
> > > we
> > > > > > could restrict vector elements to non-nested or primitive types.
> > > > > > - There are many examples of databases that restrict their vector
> > > types
> > > > > to
> > > > > > non-null, finite elements.
> > > > > > - Some people want to have write-time guarantees that vectors
> have
> > > only
> > > > > > non-null, finite elements
> > > > > > - Some people want to extend this work to include tensors,
> shorter
> > > > > floating
> > > > > > point representations, variable lengths, NaN or Infinity
> elements,
> > > etc.
> > > > > >
> > > > > > It sounds like we are nearly there, but have two halves of the
> > > > community
> > > > > > that care about different things. That just means we should have
> > > > multiple
> > > > > > types. It's entirely reasonable to want a type that allows NaN
> > > values,
> > > > > and
> > > > > > it's also reasonable for those of us building databases, table
> > > > standards,
> > > > > > and vector indexes to care about guarantees that elements are
> > finite.
> > > > > > There's more than enough examples to show that this is a
> legitimate
> > > use
> > > > > > case.
> > > > > >
> > > > > > Since I don't see the concerns of the different use cases
> changing,
> > > it
> > > > > > looks like we need to find a compromise to move forward. I
> propose
> > > that
> > > > > we
> > > > > > introduce a FINITE_VECTOR type and separately a FIXED_LENGTH_LIST
> > or
> > > > > > another VECTOR type that allows infinite and NaN values.
> > > > > >
> > > > > > Can we agree on a compromise? Is there a strong reason to try to
> > fit
> > > > > these
> > > > > > use cases into a single type? What if that means not making
> > progress?
> > > > > >
> > > > > > Ryan
> > > > > >
> > > > > > On Mon, Oct 5, 2026 at 12:15 PM Philipp Fischbeck via dev <
> > > > > > dev@parquet.apache.org> wrote:
> > > > > >
> > > > > > > Thinking about this more, I am also fine with pulling out the
> > > > “NaN/Inf”
> > > > > > > requirement. Either into a type boolean (my preference), or
> > putting
> > > > no
> > > > > > > restrictions on the Parquet level, allowing all IEEE 754 float
> > > > numbers.
> > > > > > If
> > > > > > > some top-level application wants to put additional constraints
> on
> > > the
> > > > > > data
> > > > > > > type, like Iceberg not allowing NaN/infinity, it can still do
> so
> > > one
> > > > > > layer
> > > > > > > above. It seems like this was the biggest point of discussion
> > > > > remaining.
> > > > > > > All these options work for me, and we should move forward with
> > this
> > > > > > > proposal.
> > > > > > >
> > > > > > > Concerning additional element types like bf16, they can be
> > > introduced
> > > > > as
> > > > > > > Parquet types (at least in the current LIST-based spec
> proposal).
> > > > > > >
> > > > > > > Concerning Will’s suggestion around representing VECTOR as a
> > > variable
> > > > > > > length list with a MIN/MAX_LENGTH metadata restriction: I think
> > > that
> > > > a
> > > > > > > dedicated logical VECTOR type still provides enough benefits by
> > > > > > describing
> > > > > > > the semantics of the data, and allowing to capture future
> changes
> > > > > around
> > > > > > > statistics. In that case, the length is better captured as an
> > exact
> > > > > > length
> > > > > > > for this type specifically. I’m also not sure about other use
> > cases
> > > > for
> > > > > > > MIN_LENGTH, so introducing a new type constraint system just
> for
> > > the
> > > > > > > MIN_LENGTH=MAX_LENGTH case for vectors doesn’t have enough
> value
> > in
> > > > my
> > > > > > > view.
> > > > > > >
> > > > > > > Best,
> > > > > > > Philipp
> > > > > > >
> > > > > > >
> > > > > > > On Thu, Oct 1, 2026 at 9:00 AM Will Edwards via dev <
> > > > > > > dev@parquet.apache.org>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > +1 Sebastian, a vector type suggests usage beyond normalized
> > > > > embeddings
> > > > > > > and
> > > > > > > > engines really aren't going to be able to elide all the
> checks
> > > > > anyway.
> > > > > > > >
> > > > > > > > Antoine, you suggested we explore metadata solutions too :)
> > > > > > > >
> > > > > > > > Today, a database supporting CHAR(N) or VARCHAR(n) can't
> > express
> > > > this
> > > > > > in
> > > > > > > > Parquet.  Engines can write opaque blobs they understand in
> the
> > > > > > metadata
> > > > > > > > (e.g. Arrow does) so the extra type info round-trips, but
> this
> > > > isn't
> > > > > > > > standardized, portable, nor interoperable.
> > > > > > > >
> > > > > > > > So perhaps standardizing constraint metadata or new Thrift
> > fields
> > > > (my
> > > > > > > pref)
> > > > > > > > can generically support VECTOR(N) whilst also being more
> > broadly
> > > > > useful
> > > > > > > for
> > > > > > > > other use cases and data types?
> > > > > > > >
> > > > > > > > Imagine metadata or Thrift fields to tag the MIN_LENGTH and
> > > > > MAX_LENGTH
> > > > > > > for
> > > > > > > > Parquet's variable-length physcial types.  These would count
> > > > logical
> > > > > > > units
> > > > > > > > (e.g. UTF8 code points for STRING) rather than bytes.
> > > > > > > >
> > > > > > > > Then one engine's VECTOR(N) can be written to Parquet as
> > LIST(N)
> > > > with
> > > > > > > > MIN_LENGTH = MAX_LENGTH = N and then readers - even different
> > > > > engines -
> > > > > > > can
> > > > > > > > pick up these constraints if they support doing so.  We don't
> > > > > actually
> > > > > > > need
> > > > > > > > a logical type to express a VECTOR then.
> > > > > > > >
> > > > > > > > This also allows standard SQL's CHAR(N) and VARCHAR(N) to be
> > > > > expressed
> > > > > > on
> > > > > > > > UTF8 columns etc.
> > > > > > > >
> > > > > > > > This also seems generally useful in the future as we approach
> > > > > > expressing
> > > > > > > > other tensor data shapes and types.
> > > > > > > >
> > > > > > > > This won't change the mechanics: the storage type doesn't
> > enforce
> > > > > these
> > > > > > > > constraints and engines must check them. It's previously
> > > > established
> > > > > > that
> > > > > > > > performance is just a software problem and there is no
> > > fundamental
> > > > > > > > performance advantage to the fixed size types in Dremel-like
> > > > formats
> > > > > > like
> > > > > > > > Parquet.
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Will
> > > > > > > >
> > > > > > > > On Wed, 30 Sept 2026 at 16:08, Sebastian Baunsgaard <
> > > > > > > baunsgaard@apache.org
> > > > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > After listening in on the last meeting Wednesday, I have
> been
> > > > > > thinking
> > > > > > > > > about this quite a bit, and thought I would share my 2
> cents.
> > > > > > > > >
> > > > > > > > > I suggest not requiring VECTOR elements to be finite, as
> > > proposed
> > > > > in
> > > > > > > the
> > > > > > > > PR
> > > > > > > > > <https://github.com/apache/parquet-format/pull/624>.
> > > > > > > > >
> > > > > > > > > While Divjot and others note the specialization for UTF-8,
> > > > Parquet
> > > > > > > > already
> > > > > > > > > defines DOUBLE and FLOAT as IEEE encodings
> > > > > > > > > <
> > > > > >
> > https://github.com/apache/parquet-format/blob/master/README.md#types
> > > > > > > >.
> > > > > > > > >
> > > > > > > > > Making NaN or +/-Inf illegal inside VECTOR would be a new
> > > > > restriction
> > > > > > > on
> > > > > > > > > those same underlying physical types. Scalar FLOAT/DOUBLE
> > > columns
> > > > > > have
> > > > > > > > > never had such a rule.
> > > > > > > > >
> > > > > > > > > It would also not match how the systems that actually
> produce
> > > > these
> > > > > > > > vectors
> > > > > > > > > treat floats. Most of them use IEEE floats, e.g. Spark,
> > DuckDB,
> > > > > > NumPy,
> > > > > > > > > PyTorch, TensorFlow, or lower level CUDA, oneMKL, and
> > OpenBLAS.
> > > > > > > > >
> > > > > > > > > pgvector's "all elements must be finite" rule is an insert
> > > > policy.
> > > > > > > While
> > > > > > > > I
> > > > > > > > > think Parquet as a general-purpose format should allow them
> > out
> > > > of
> > > > > > the
> > > > > > > > box.
> > > > > > > > > It is a user system decision to disallow writing these
> values
> > > or
> > > > to
> > > > > > > throw
> > > > > > > > > errors when encountering them.
> > > > > > > > >
> > > > > > > > > A type-level ban also does not keep Inf/NaN out of vector
> > math.
> > > > > > Finite
> > > > > > > > > coordinates still overflow. For instance: magnitude =
> > > > ||[FLOAT_MAX,
> > > > > > > > > FLOAT_MAX]|| where every coordinate is finite overflows to
> > > +Inf.
> > > > > > > PyTorch
> > > > > > > > > documents the same overflow: tensor([1e20, 1e20]).norm() is
> > inf
> > > > in
> > > > > > > fp32.
> > > > > > > > >
> > > > > > > > > A generated MAGNITUDE(vec) column would then be Inf while
> the
> > > > > VECTOR
> > > > > > > > itself
> > > > > > > > > was required to be finite. The ban only forbids storing
> > > specials
> > > > in
> > > > > > the
> > > > > > > > > coordinates, and does not prevent even simple downstream
> > > > > computations
> > > > > > > > from
> > > > > > > > > producing them.
> > > > > > > > >
> > > > > > > > > I suggest keeping VECTOR as the type and add an optional
> bool
> > > > > > > > > VectorType.finite (default false). Engines that want
> > > > pgvector-like
> > > > > > > > > rejection can require the flag = true. Engines that use
> > NumPy /
> > > > > > > PyTorch /
> > > > > > > > > Spark arrays can keep representable IEEE values. For the
> same
> > > > > > reason, I
> > > > > > > > do
> > > > > > > > > not favor adding yet another type called FINITE_VECTOR.
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > > Sebastian
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Tue, Sep 29, 2026 at 2:03 PM Divjot Arora via dev <
> > > > > > > > > dev@parquet.apache.org>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > I agree with moving forward with the proposal and
> renaming
> > > > VECTOR
> > > > > > to
> > > > > > > > > > FINITE_VECTOR or something
> > > > > > > > > > similar if others feel that better communicates its
> > intent. I
> > > > see
> > > > > > > value
> > > > > > > > > in
> > > > > > > > > > having a vector type that guarantees
> > > > > > > > > > its values are mathematically-correct vectors for a few
> > > > reasons:
> > > > > > > > > >
> > > > > > > > > > 1. As others have mentioned, this is similar to the
> STRING
> > > > > logical
> > > > > > > type
> > > > > > > > > > mandating UTF-8.
> > > > > > > > > > 2. It is not possible to rely on statistics for detecting
> > NaN
> > > > or
> > > > > > > +-inf
> > > > > > > > > > values as all statistics are optional.
> > > > > > > > > > To my knowledge, there is no precedent for requiring that
> > > > writers
> > > > > > > emit
> > > > > > > > > > stats for specific logical types.
> > > > > > > > > > 3. Without statistics, NaN/inf values could flow through
> > the
> > > > > engine
> > > > > > > and
> > > > > > > > > > block them for specific functionality.
> > > > > > > > > > This seems to be Lance's behavior. However, I don't think
> > > this
> > > > is
> > > > > > as
> > > > > > > > easy
> > > > > > > > > > as it sounds and Lance has run
> > > > > > > > > > into several bugs due to this ([1] [2] for examples).
> Their
> > > > > Python
> > > > > > > [3]
> > > > > > > > > and
> > > > > > > > > > Rust [4] APIs default to erroring on
> > > > > > > > > > vectors that contain NaN values.
> > > > > > > > > >
> > > > > > > > > > I agree that Parquet is a general-purpose format, but I
> > don't
> > > > > think
> > > > > > > > this
> > > > > > > > > > implies the lack of domain-specific types.
> > > > > > > > > > Instead, this broadens the use cases where Parquet would
> be
> > > > > > > considered
> > > > > > > > a
> > > > > > > > > > good fit. I also fully agree with
> > > > > > > > > > continuing work on a general fixed-size list type if we
> > > believe
> > > > > > there
> > > > > > > > are
> > > > > > > > > > non-vector use cases that would benefit
> > > > > > > > > > from such guarantees at the storage layer.
> > > > > > > > > >
> > > > > > > > > > [1] https://github.com/lance-format/lance/pull/3648
> > > > > > > > > > [2] https://github.com/lance-format/lance/issues/8607
> (see
> > > > > point 2
> > > > > > > in
> > > > > > > > > > "root
> > > > > > > > > > cause")
> > > > > > > > > > [3]
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/lancedb/lancedb/blob/0327618a62483feb6dca2336549845e5887cd48f/python/python/lancedb/table.py#L4807-L4911
> > > > > > > > > > [4]
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/lancedb/lancedb/blob/0327618a62483feb6dca2336549845e5887cd48f/rust/lancedb/src/table/add_data.rs#L105-L110
> > > > > > > > > >
> > > > > > > > > > On Mon, Sep 28, 2026 at 6:19 PM Philipp Fischbeck via
> dev <
> > > > > > > > > > dev@parquet.apache.org> wrote:
> > > > > > > > > >
> > > > > > > > > > > I also favor moving forward with the proposal as is. I
> > see
> > > > > strong
> > > > > > > > value
> > > > > > > > > > and
> > > > > > > > > > > use cases for the numeric vector type, and I agree with
> > the
> > > > > > > > > restrictions
> > > > > > > > > > > around NaN/Inf values and element types in general.
> > > Precedent
> > > > > > > exists
> > > > > > > > in
> > > > > > > > > > > other types that restrict which physical type values
> are
> > > > > > logically
> > > > > > > > > valid
> > > > > > > > > > > (e.g., the geospatial types needing to be WKB-encoded).
> > A
> > > > > > general
> > > > > > > > > > > fixed-size list also has value, but I think it makes
> > sense
> > > to
> > > > > > > > separate
> > > > > > > > > > the
> > > > > > > > > > > two types. The semantics are relevant, and I can see
> how
> > > the
> > > > > two
> > > > > > > > types
> > > > > > > > > > > might diverge further in the future regarding
> statistics
> > or
> > > > > > > > encodings.
> > > > > > > > > > >
> > > > > > > > > > > I would be okay with renaming the vector type, but it
> > seems
> > > > > like
> > > > > > > the
> > > > > > > > > > VECTOR
> > > > > > > > > > > name is already used across other engines mentioned in
> > this
> > > > > > thread.
> > > > > > > > > > >
> > > > > > > > > > > Best,
> > > > > > > > > > > Philipp
> > > > > > > > > > >
> > > > > > > > > > > On Sat, Sep 26, 2026 at 4:37 PM Gang Wu <
> > ustcwg@gmail.com>
> > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > To separate orthogonal concerns, I feel we should
> treat
> > > the
> > > > > > > general
> > > > > > > > > > > > FixedSizeList as the underlying physical layout, and
> > > > separate
> > > > > > it
> > > > > > > > from
> > > > > > > > > > the
> > > > > > > > > > > > discussion of a specialized VECTOR (or broader ML
> > tensor)
> > > > > > logical
> > > > > > > > > type.
> > > > > > > > > > > >
> > > > > > > > > > > > Currently, the immediate need is to define a
> > specialized
> > > > > VECTOR
> > > > > > > > type
> > > > > > > > > so
> > > > > > > > > > > > that different engines and formats have a common
> > semantic
> > > > > > > contract
> > > > > > > > > for
> > > > > > > > > > > > interoperability.
> > > > > > > > > > > >
> > > > > > > > > > > > If the main point of contention is just the
> > NaN/Inf/null
> > > > > > > > limitations,
> > > > > > > > > > > maybe
> > > > > > > > > > > > we can resolve this by adding a qualification to the
> > name
> > > > > (like
> > > > > > > > > Andrew
> > > > > > > > > > > > suggested, e.g., VECTOR_EMBEDDING or FINITE_VECTOR)
> to
> > > > avoid
> > > > > > > > > confusion
> > > > > > > > > > > with
> > > > > > > > > > > > broader ML use cases. Alternatively, we could
> > introduce a
> > > > > > boolean
> > > > > > > > > flag
> > > > > > > > > > > > (e.g., optional bool VectorType.finite) as proposed
> > > > earlier.
> > > > > > > > > > > >
> > > > > > > > > > > > On Fri, Sep 25, 2026 at 3:52 AM Andrew Lamb <
> > > > > > > > andrewlamb11@gmail.com>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > I am a little confused about the points of
> > disagreement
> > > > > here.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Is it that the name `VECTOR` is too general for the
> > > > > specific
> > > > > > > use
> > > > > > > > > case
> > > > > > > > > > > of
> > > > > > > > > > > > > fixed length arrays with non-finite values used for
> > > > vector
> > > > > > > > > > embeddings?
> > > > > > > > > > > > If
> > > > > > > > > > > > > so, maybe we could add a qualification to the name
> > > > > > > > (FINITE_VECTOR?
> > > > > > > > > > > > > FEATURE_VECTOR? VECTOR_EMBEDDING?)
> > > > > > > > > > > > >
> > > > > > > > > > > > > It seems like there is precedent for defining
> logical
> > > > types
> > > > > > > that
> > > > > > > > > > > specify
> > > > > > > > > > > > > certain physical types and restrict their contents
> to
> > > > help
> > > > > > > > readers
> > > > > > > > > > > > > interpret meaning, for example string[1] or JSON;
> > > > Defining
> > > > > > one
> > > > > > > > for
> > > > > > > > > > > > vector
> > > > > > > > > > > > > embeddings seems reasonable enough to me.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Andrew
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > [1]:
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#string
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Thu, Sep 24, 2026 at 3:34 PM Daniel Weeks <
> > > > > > > dweeks@apache.org>
> > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Well, Parquet is not a single-purpose format.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Vector stores can choose whatever policy fits
> > their
> > > > > > usage,
> > > > > > > > but
> > > > > > > > > > > we're
> > > > > > > > > > > > > > > talking about a general-purpose file format
> that
> > > aims
> > > > > to
> > > > > > be
> > > > > > > > > > broadly
> > > > > > > > > > > > > > > applicable.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I agree that the format is intended to be general
> > > > > purpose,
> > > > > > > but
> > > > > > > > > > types
> > > > > > > > > > > > > within
> > > > > > > > > > > > > > the format define specific representation and
> > > semantics
> > > > > to
> > > > > > > > > achieve
> > > > > > > > > > > that
> > > > > > > > > > > > > > goal.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > "Vector" has a more general sense than ML and
> > it's
> > > > > quite
> > > > > > > > > trivial
> > > > > > > > > > to
> > > > > > > > > > > > > > > produce vectors with non-finite values using
> e.g.
> > > > NumPy
> > > > > > or
> > > > > > > > > Pandas
> > > > > > > > > > > (or
> > > > > > > > > > > > > > > any other piece of dataframe or scientific
> > > computing
> > > > > > > > software).
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >  The justification for restricting non-finite
> > values
> > > is
> > > > > > > > directly
> > > > > > > > > > > > related
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > the semantic meaning of a vector.  Vectors have
> > > > specific
> > > > > > > > > > > > characteristics
> > > > > > > > > > > > > > (dimension, magnitude, direction, etc).
> Non-finite
> > > > > values
> > > > > > > are
> > > > > > > > > > > invalid
> > > > > > > > > > > > > and
> > > > > > > > > > > > > > break those characteristics.  I'd liken it to
> > storing
> > > > > > > non-utf8
> > > > > > > > > > values
> > > > > > > > > > > > in
> > > > > > > > > > > > > a
> > > > > > > > > > > > > > string and still calling it a string.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > That also differentiates a vector from a fixed
> > length
> > > > > list,
> > > > > > > > which
> > > > > > > > > > is
> > > > > > > > > > > > more
> > > > > > > > > > > > > > general and does not carry the same semantic
> > meaning.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I also don't think we need to bundle all
> similarly
> > > > shaped
> > > > > > use
> > > > > > > > > cases
> > > > > > > > > > > > into
> > > > > > > > > > > > > > the same solution (e.g. flxed size list).  Vector
> > is
> > > a
> > > > > well
> > > > > > > > > > > established
> > > > > > > > > > > > > > data type at this point and aligns with the
> general
> > > > > purpose
> > > > > > > > > nature
> > > > > > > > > > of
> > > > > > > > > > > > the
> > > > > > > > > > > > > > format (many other formats and databases natively
> > > > support
> > > > > > > > this).
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > -Dan
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Thu, Sep 24, 2026 at 11:55 AM Antoine Pitrou <
> > > > > > > > > > antoine@python.org>
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Le 24/09/2026 à 20:11, Russell Spitzer a écrit
> :
> > > > > > > > > > > > > > > > I think something I haven't seen yet is the
> > clear
> > > > > > > use-case
> > > > > > > > > for
> > > > > > > > > > > > > > non-finite
> > > > > > > > > > > > > > > > values within a vector type. There has been a
> > lot
> > > > of
> > > > > > > > > > theoretical
> > > > > > > > > > > > > > > discussion
> > > > > > > > > > > > > > > > here about use cases but what toolings
> > currently
> > > > > > produce
> > > > > > > > > these
> > > > > > > > > > > > > vectors
> > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > how do they function in downstream engines?
> I'd
> > > > like
> > > > > to
> > > > > > > > limit
> > > > > > > > > > > > > > introducing
> > > > > > > > > > > > > > > > types that have a wide range of values which
> > are
> > > > > simply
> > > > > > > are
> > > > > > > > > > > failure
> > > > > > > > > > > > > > modes
> > > > > > > > > > > > > > > > of different flavors
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > "Vector" has a more general sense than ML and
> > it's
> > > > > quite
> > > > > > > > > trivial
> > > > > > > > > > to
> > > > > > > > > > > > > > > produce vectors with non-finite values using
> e.g.
> > > > NumPy
> > > > > > or
> > > > > > > > > Pandas
> > > > > > > > > > > (or
> > > > > > > > > > > > > > > any other piece of dataframe or scientific
> > > computing
> > > > > > > > software).
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > It's not like the general concept of a vector
> is
> > a
> > > > IETF
> > > > > > > > > standard
> > > > > > > > > > or
> > > > > > > > > > > > > > > anything, so analogies with JSON or UTF8 (where
> > the
> > > > > > values
> > > > > > > > are
> > > > > > > > > > > > expected
> > > > > > > > > > > > > > > to be valid according to the respective specs)
> do
> > > not
> > > > > > > apply.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > This discussion would IMHO have a better chance
> > of
> > > > > > > converging
> > > > > > > > > if
> > > > > > > > > > we
> > > > > > > > > > > > > > > separated the two orthogonal concerns of:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 1. representing fixed-length list/vector data
> in
> > > > > Parquet
> > > > > > > > (with
> > > > > > > > > a
> > > > > > > > > > > > > > > potential limitation to non-nested types)
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 2. representing the information that a given
> > column
> > > > (or
> > > > > > > > column
> > > > > > > > > > > chunk,
> > > > > > > > > > > > > > > etc.) does not contain non-finite values
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > ... both of which are useful on their own
> right,
> > > but
> > > > do
> > > > > > not
> > > > > > > > > have
> > > > > > > > > > to
> > > > > > > > > > > > be
> > > > > > > > > > > > > > > conflated and bundled together.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Regards
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Antoine.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] Numeric Vector type: physical representation (outcome of focused call)

[Gang Wu <us...@gmail.com>]

I think separating them into FINITE_VECTOR and FIXED_LENGTH_LIST is a good
idea, though I think renaming FINITE_VECTOR to VECTOR looks more natural.
Should we move forward with proposing FINITE_VECTOR/VECTOR first?

On Tue, Oct 13, 2026 at 6:26 PM Alkis Evlogimenos via dev <
dev@parquet.apache.org> wrote:

> I suggest we go with VECTOR for the finite vector to keep the name the same
> as other engines/type systems. If we want a non-finite one in the future we
> can name that something else or we add fixed size list that will serve the
> same purpose without a special name.
>
>
> On Thu, Oct 8, 2026 at 10:20 PM Ryan Blue <bl...@apache.org> wrote:
>
> > A logical type would:
> > - Describe data for users and readers to understand what the file
> contains
> > - Signal that repetition and definition levels will reflect a fixed
> number
> > of elements, which implementations can use to optimize decoding
> > - Guarantee that there are no null elements (because the type cannot be
> > applied to a list with optional elements)
> > - [FINITE_VECTOR] Guarantee that no elements are NaN or Infinity values
> so
> > that the vectors have a finite length and are suitable for distance
> > calculations
> >
> > The finite value restriction is a restriction on writers to prevent bad
> > data from being written.
> >
> > Ignoring logical types that are not understood is a rule for readers. A
> > reader can handle vectors and pass it on to another layer. For instance,
> a
> > Parquet reader that doesn't support geometry passes values as `binary`.
> > That binary value can be passed to a geometry UDF, even if the underlying
> > engine and Parquet reader doesn't have support.
> >
> > A tool that doesn't understand FINITE_VECTOR can similarly pass a list of
> > floats along to a UDF. But it is still important to the user that bad
> > values are not present in the data. In your example, bad values are
> vectors
> > that cause the centroid calculation to fail, like vectors with infinite
> > length.
> >
> > Ryan
> >
> > On Thu, Oct 8, 2026 at 12:54 PM Will Edwards via dev <
> > dev@parquet.apache.org>
> > wrote:
> >
> > > Could we clarify the use cases for a VECTOR annotation on LIST<FLOAT>?
> > > Specifically, what does it make possible that's impossible today, and
> > what
> > > does it make easier that's currently hard?
> > >
> > > I'm struggling to find a strong answer. I'd like every engine to offer
> > > vector functions on lists of floats without needing a new logical type.
> > >
> > > To be clear, I'm still enthusiastic about a vector type that describes
> > how
> > > to unpack and scale composite representations (quantised codes plus
> > scales,
> > > codebooks, etc.), in the direction ML is going. That adds information
> > > engines don't have. A label on LIST<FLOAT> doesn't: engines already
> > > understand the physical type, and many already offer vector functions
> on
> > > it. I don't see a performance benefit, and NaN/null/Inf checks are
> cheap
> > > enough that engines should do them anyway. The label also doesn't say
> > what
> > > you can do with the column. Knowing that a column holds embeddings
> > doesn't
> > > say how they can be used, just as zipcode and age are both integers but
> > > multiplying them is meaningless.
> > >
> > > My concern is the opposite effect. We're establishing that engines
> should
> > > ignore logical types they don't understand. So a tool that doesn't know
> > > VECTOR will read the column as LIST<FLOAT>, let the user compute
> > centroids
> > > (say), and write plain LIST<FLOAT> back out. If other engines restrict
> > > embedding functions to VECTOR-annotated columns, that file stops
> working
> > > with them, through no fault of the user. An annotation that doesn't
> > change
> > > what the data is shouldn't be able to make it unusable.
> > >
> > > Am I missing a use case?
> > >
> > > Best, Will
> > >
> > > On Thu, 8 Oct 2026 at 19:53, Steven Wu <st...@gmail.com> wrote:
> > >
> > > > I also think a FINITE_VECTOR type brings a lot of values to many use
> > > cases.
> > > > It is would be great to make progress on the FINITE_VECTOR, and not
> to
> > > fit
> > > > every use cases into the FIXED_LENGTH_LIST.
> > > >
> > > > On Mon, Oct 5, 2026 at 1:26 PM Ryan Blue <bl...@apache.org> wrote:
> > > >
> > > > > Going through this thread again, it looks like there are a few
> things
> > > > that
> > > > > people generally agree on:
> > > > >
> > > > > - In the short term, we want to introduce a logical type or types;
> a
> > > > better
> > > > > physical representation can be done later. We can focus on logical
> > > types.
> > > > > - There is precedent in logical types for restricting values, like
> > UTF8
> > > > > strings to contain only UTF-8 codepoints. I think we also agree
> that
> > we
> > > > > could restrict vector elements to non-nested or primitive types.
> > > > > - There are many examples of databases that restrict their vector
> > types
> > > > to
> > > > > non-null, finite elements.
> > > > > - Some people want to have write-time guarantees that vectors have
> > only
> > > > > non-null, finite elements
> > > > > - Some people want to extend this work to include tensors, shorter
> > > > floating
> > > > > point representations, variable lengths, NaN or Infinity elements,
> > etc.
> > > > >
> > > > > It sounds like we are nearly there, but have two halves of the
> > > community
> > > > > that care about different things. That just means we should have
> > > multiple
> > > > > types. It's entirely reasonable to want a type that allows NaN
> > values,
> > > > and
> > > > > it's also reasonable for those of us building databases, table
> > > standards,
> > > > > and vector indexes to care about guarantees that elements are
> finite.
> > > > > There's more than enough examples to show that this is a legitimate
> > use
> > > > > case.
> > > > >
> > > > > Since I don't see the concerns of the different use cases changing,
> > it
> > > > > looks like we need to find a compromise to move forward. I propose
> > that
> > > > we
> > > > > introduce a FINITE_VECTOR type and separately a FIXED_LENGTH_LIST
> or
> > > > > another VECTOR type that allows infinite and NaN values.
> > > > >
> > > > > Can we agree on a compromise? Is there a strong reason to try to
> fit
> > > > these
> > > > > use cases into a single type? What if that means not making
> progress?
> > > > >
> > > > > Ryan
> > > > >
> > > > > On Mon, Oct 5, 2026 at 12:15 PM Philipp Fischbeck via dev <
> > > > > dev@parquet.apache.org> wrote:
> > > > >
> > > > > > Thinking about this more, I am also fine with pulling out the
> > > “NaN/Inf”
> > > > > > requirement. Either into a type boolean (my preference), or
> putting
> > > no
> > > > > > restrictions on the Parquet level, allowing all IEEE 754 float
> > > numbers.
> > > > > If
> > > > > > some top-level application wants to put additional constraints on
> > the
> > > > > data
> > > > > > type, like Iceberg not allowing NaN/infinity, it can still do so
> > one
> > > > > layer
> > > > > > above. It seems like this was the biggest point of discussion
> > > > remaining.
> > > > > > All these options work for me, and we should move forward with
> this
> > > > > > proposal.
> > > > > >
> > > > > > Concerning additional element types like bf16, they can be
> > introduced
> > > > as
> > > > > > Parquet types (at least in the current LIST-based spec proposal).
> > > > > >
> > > > > > Concerning Will’s suggestion around representing VECTOR as a
> > variable
> > > > > > length list with a MIN/MAX_LENGTH metadata restriction: I think
> > that
> > > a
> > > > > > dedicated logical VECTOR type still provides enough benefits by
> > > > > describing
> > > > > > the semantics of the data, and allowing to capture future changes
> > > > around
> > > > > > statistics. In that case, the length is better captured as an
> exact
> > > > > length
> > > > > > for this type specifically. I’m also not sure about other use
> cases
> > > for
> > > > > > MIN_LENGTH, so introducing a new type constraint system just for
> > the
> > > > > > MIN_LENGTH=MAX_LENGTH case for vectors doesn’t have enough value
> in
> > > my
> > > > > > view.
> > > > > >
> > > > > > Best,
> > > > > > Philipp
> > > > > >
> > > > > >
> > > > > > On Thu, Oct 1, 2026 at 9:00 AM Will Edwards via dev <
> > > > > > dev@parquet.apache.org>
> > > > > > wrote:
> > > > > >
> > > > > > > +1 Sebastian, a vector type suggests usage beyond normalized
> > > > embeddings
> > > > > > and
> > > > > > > engines really aren't going to be able to elide all the checks
> > > > anyway.
> > > > > > >
> > > > > > > Antoine, you suggested we explore metadata solutions too :)
> > > > > > >
> > > > > > > Today, a database supporting CHAR(N) or VARCHAR(n) can't
> express
> > > this
> > > > > in
> > > > > > > Parquet.  Engines can write opaque blobs they understand in the
> > > > > metadata
> > > > > > > (e.g. Arrow does) so the extra type info round-trips, but this
> > > isn't
> > > > > > > standardized, portable, nor interoperable.
> > > > > > >
> > > > > > > So perhaps standardizing constraint metadata or new Thrift
> fields
> > > (my
> > > > > > pref)
> > > > > > > can generically support VECTOR(N) whilst also being more
> broadly
> > > > useful
> > > > > > for
> > > > > > > other use cases and data types?
> > > > > > >
> > > > > > > Imagine metadata or Thrift fields to tag the MIN_LENGTH and
> > > > MAX_LENGTH
> > > > > > for
> > > > > > > Parquet's variable-length physcial types.  These would count
> > > logical
> > > > > > units
> > > > > > > (e.g. UTF8 code points for STRING) rather than bytes.
> > > > > > >
> > > > > > > Then one engine's VECTOR(N) can be written to Parquet as
> LIST(N)
> > > with
> > > > > > > MIN_LENGTH = MAX_LENGTH = N and then readers - even different
> > > > engines -
> > > > > > can
> > > > > > > pick up these constraints if they support doing so.  We don't
> > > > actually
> > > > > > need
> > > > > > > a logical type to express a VECTOR then.
> > > > > > >
> > > > > > > This also allows standard SQL's CHAR(N) and VARCHAR(N) to be
> > > > expressed
> > > > > on
> > > > > > > UTF8 columns etc.
> > > > > > >
> > > > > > > This also seems generally useful in the future as we approach
> > > > > expressing
> > > > > > > other tensor data shapes and types.
> > > > > > >
> > > > > > > This won't change the mechanics: the storage type doesn't
> enforce
> > > > these
> > > > > > > constraints and engines must check them. It's previously
> > > established
> > > > > that
> > > > > > > performance is just a software problem and there is no
> > fundamental
> > > > > > > performance advantage to the fixed size types in Dremel-like
> > > formats
> > > > > like
> > > > > > > Parquet.
> > > > > > >
> > > > > > > Best,
> > > > > > > Will
> > > > > > >
> > > > > > > On Wed, 30 Sept 2026 at 16:08, Sebastian Baunsgaard <
> > > > > > baunsgaard@apache.org
> > > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > After listening in on the last meeting Wednesday, I have been
> > > > > thinking
> > > > > > > > about this quite a bit, and thought I would share my 2 cents.
> > > > > > > >
> > > > > > > > I suggest not requiring VECTOR elements to be finite, as
> > proposed
> > > > in
> > > > > > the
> > > > > > > PR
> > > > > > > > <https://github.com/apache/parquet-format/pull/624>.
> > > > > > > >
> > > > > > > > While Divjot and others note the specialization for UTF-8,
> > > Parquet
> > > > > > > already
> > > > > > > > defines DOUBLE and FLOAT as IEEE encodings
> > > > > > > > <
> > > > >
> https://github.com/apache/parquet-format/blob/master/README.md#types
> > > > > > >.
> > > > > > > >
> > > > > > > > Making NaN or +/-Inf illegal inside VECTOR would be a new
> > > > restriction
> > > > > > on
> > > > > > > > those same underlying physical types. Scalar FLOAT/DOUBLE
> > columns
> > > > > have
> > > > > > > > never had such a rule.
> > > > > > > >
> > > > > > > > It would also not match how the systems that actually produce
> > > these
> > > > > > > vectors
> > > > > > > > treat floats. Most of them use IEEE floats, e.g. Spark,
> DuckDB,
> > > > > NumPy,
> > > > > > > > PyTorch, TensorFlow, or lower level CUDA, oneMKL, and
> OpenBLAS.
> > > > > > > >
> > > > > > > > pgvector's "all elements must be finite" rule is an insert
> > > policy.
> > > > > > While
> > > > > > > I
> > > > > > > > think Parquet as a general-purpose format should allow them
> out
> > > of
> > > > > the
> > > > > > > box.
> > > > > > > > It is a user system decision to disallow writing these values
> > or
> > > to
> > > > > > throw
> > > > > > > > errors when encountering them.
> > > > > > > >
> > > > > > > > A type-level ban also does not keep Inf/NaN out of vector
> math.
> > > > > Finite
> > > > > > > > coordinates still overflow. For instance: magnitude =
> > > ||[FLOAT_MAX,
> > > > > > > > FLOAT_MAX]|| where every coordinate is finite overflows to
> > +Inf.
> > > > > > PyTorch
> > > > > > > > documents the same overflow: tensor([1e20, 1e20]).norm() is
> inf
> > > in
> > > > > > fp32.
> > > > > > > >
> > > > > > > > A generated MAGNITUDE(vec) column would then be Inf while the
> > > > VECTOR
> > > > > > > itself
> > > > > > > > was required to be finite. The ban only forbids storing
> > specials
> > > in
> > > > > the
> > > > > > > > coordinates, and does not prevent even simple downstream
> > > > computations
> > > > > > > from
> > > > > > > > producing them.
> > > > > > > >
> > > > > > > > I suggest keeping VECTOR as the type and add an optional bool
> > > > > > > > VectorType.finite (default false). Engines that want
> > > pgvector-like
> > > > > > > > rejection can require the flag = true. Engines that use
> NumPy /
> > > > > > PyTorch /
> > > > > > > > Spark arrays can keep representable IEEE values. For the same
> > > > > reason, I
> > > > > > > do
> > > > > > > > not favor adding yet another type called FINITE_VECTOR.
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Sebastian
> > > > > > > >
> > > > > > > >
> > > > > > > > On Tue, Sep 29, 2026 at 2:03 PM Divjot Arora via dev <
> > > > > > > > dev@parquet.apache.org>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > I agree with moving forward with the proposal and renaming
> > > VECTOR
> > > > > to
> > > > > > > > > FINITE_VECTOR or something
> > > > > > > > > similar if others feel that better communicates its
> intent. I
> > > see
> > > > > > value
> > > > > > > > in
> > > > > > > > > having a vector type that guarantees
> > > > > > > > > its values are mathematically-correct vectors for a few
> > > reasons:
> > > > > > > > >
> > > > > > > > > 1. As others have mentioned, this is similar to the STRING
> > > > logical
> > > > > > type
> > > > > > > > > mandating UTF-8.
> > > > > > > > > 2. It is not possible to rely on statistics for detecting
> NaN
> > > or
> > > > > > +-inf
> > > > > > > > > values as all statistics are optional.
> > > > > > > > > To my knowledge, there is no precedent for requiring that
> > > writers
> > > > > > emit
> > > > > > > > > stats for specific logical types.
> > > > > > > > > 3. Without statistics, NaN/inf values could flow through
> the
> > > > engine
> > > > > > and
> > > > > > > > > block them for specific functionality.
> > > > > > > > > This seems to be Lance's behavior. However, I don't think
> > this
> > > is
> > > > > as
> > > > > > > easy
> > > > > > > > > as it sounds and Lance has run
> > > > > > > > > into several bugs due to this ([1] [2] for examples). Their
> > > > Python
> > > > > > [3]
> > > > > > > > and
> > > > > > > > > Rust [4] APIs default to erroring on
> > > > > > > > > vectors that contain NaN values.
> > > > > > > > >
> > > > > > > > > I agree that Parquet is a general-purpose format, but I
> don't
> > > > think
> > > > > > > this
> > > > > > > > > implies the lack of domain-specific types.
> > > > > > > > > Instead, this broadens the use cases where Parquet would be
> > > > > > considered
> > > > > > > a
> > > > > > > > > good fit. I also fully agree with
> > > > > > > > > continuing work on a general fixed-size list type if we
> > believe
> > > > > there
> > > > > > > are
> > > > > > > > > non-vector use cases that would benefit
> > > > > > > > > from such guarantees at the storage layer.
> > > > > > > > >
> > > > > > > > > [1] https://github.com/lance-format/lance/pull/3648
> > > > > > > > > [2] https://github.com/lance-format/lance/issues/8607 (see
> > > > point 2
> > > > > > in
> > > > > > > > > "root
> > > > > > > > > cause")
> > > > > > > > > [3]
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/lancedb/lancedb/blob/0327618a62483feb6dca2336549845e5887cd48f/python/python/lancedb/table.py#L4807-L4911
> > > > > > > > > [4]
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/lancedb/lancedb/blob/0327618a62483feb6dca2336549845e5887cd48f/rust/lancedb/src/table/add_data.rs#L105-L110
> > > > > > > > >
> > > > > > > > > On Mon, Sep 28, 2026 at 6:19 PM Philipp Fischbeck via dev <
> > > > > > > > > dev@parquet.apache.org> wrote:
> > > > > > > > >
> > > > > > > > > > I also favor moving forward with the proposal as is. I
> see
> > > > strong
> > > > > > > value
> > > > > > > > > and
> > > > > > > > > > use cases for the numeric vector type, and I agree with
> the
> > > > > > > > restrictions
> > > > > > > > > > around NaN/Inf values and element types in general.
> > Precedent
> > > > > > exists
> > > > > > > in
> > > > > > > > > > other types that restrict which physical type values are
> > > > > logically
> > > > > > > > valid
> > > > > > > > > > (e.g., the geospatial types needing to be WKB-encoded).
> A
> > > > > general
> > > > > > > > > > fixed-size list also has value, but I think it makes
> sense
> > to
> > > > > > > separate
> > > > > > > > > the
> > > > > > > > > > two types. The semantics are relevant, and I can see how
> > the
> > > > two
> > > > > > > types
> > > > > > > > > > might diverge further in the future regarding statistics
> or
> > > > > > > encodings.
> > > > > > > > > >
> > > > > > > > > > I would be okay with renaming the vector type, but it
> seems
> > > > like
> > > > > > the
> > > > > > > > > VECTOR
> > > > > > > > > > name is already used across other engines mentioned in
> this
> > > > > thread.
> > > > > > > > > >
> > > > > > > > > > Best,
> > > > > > > > > > Philipp
> > > > > > > > > >
> > > > > > > > > > On Sat, Sep 26, 2026 at 4:37 PM Gang Wu <
> ustcwg@gmail.com>
> > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > To separate orthogonal concerns, I feel we should treat
> > the
> > > > > > general
> > > > > > > > > > > FixedSizeList as the underlying physical layout, and
> > > separate
> > > > > it
> > > > > > > from
> > > > > > > > > the
> > > > > > > > > > > discussion of a specialized VECTOR (or broader ML
> tensor)
> > > > > logical
> > > > > > > > type.
> > > > > > > > > > >
> > > > > > > > > > > Currently, the immediate need is to define a
> specialized
> > > > VECTOR
> > > > > > > type
> > > > > > > > so
> > > > > > > > > > > that different engines and formats have a common
> semantic
> > > > > > contract
> > > > > > > > for
> > > > > > > > > > > interoperability.
> > > > > > > > > > >
> > > > > > > > > > > If the main point of contention is just the
> NaN/Inf/null
> > > > > > > limitations,
> > > > > > > > > > maybe
> > > > > > > > > > > we can resolve this by adding a qualification to the
> name
> > > > (like
> > > > > > > > Andrew
> > > > > > > > > > > suggested, e.g., VECTOR_EMBEDDING or FINITE_VECTOR) to
> > > avoid
> > > > > > > > confusion
> > > > > > > > > > with
> > > > > > > > > > > broader ML use cases. Alternatively, we could
> introduce a
> > > > > boolean
> > > > > > > > flag
> > > > > > > > > > > (e.g., optional bool VectorType.finite) as proposed
> > > earlier.
> > > > > > > > > > >
> > > > > > > > > > > On Fri, Sep 25, 2026 at 3:52 AM Andrew Lamb <
> > > > > > > andrewlamb11@gmail.com>
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > I am a little confused about the points of
> disagreement
> > > > here.
> > > > > > > > > > > >
> > > > > > > > > > > > Is it that the name `VECTOR` is too general for the
> > > > specific
> > > > > > use
> > > > > > > > case
> > > > > > > > > > of
> > > > > > > > > > > > fixed length arrays with non-finite values used for
> > > vector
> > > > > > > > > embeddings?
> > > > > > > > > > > If
> > > > > > > > > > > > so, maybe we could add a qualification to the name
> > > > > > > (FINITE_VECTOR?
> > > > > > > > > > > > FEATURE_VECTOR? VECTOR_EMBEDDING?)
> > > > > > > > > > > >
> > > > > > > > > > > > It seems like there is precedent for defining logical
> > > types
> > > > > > that
> > > > > > > > > > specify
> > > > > > > > > > > > certain physical types and restrict their contents to
> > > help
> > > > > > > readers
> > > > > > > > > > > > interpret meaning, for example string[1] or JSON;
> > > Defining
> > > > > one
> > > > > > > for
> > > > > > > > > > > vector
> > > > > > > > > > > > embeddings seems reasonable enough to me.
> > > > > > > > > > > >
> > > > > > > > > > > > Andrew
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > [1]:
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#string
> > > > > > > > > > > >
> > > > > > > > > > > > On Thu, Sep 24, 2026 at 3:34 PM Daniel Weeks <
> > > > > > dweeks@apache.org>
> > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Well, Parquet is not a single-purpose format.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Vector stores can choose whatever policy fits
> their
> > > > > usage,
> > > > > > > but
> > > > > > > > > > we're
> > > > > > > > > > > > > > talking about a general-purpose file format that
> > aims
> > > > to
> > > > > be
> > > > > > > > > broadly
> > > > > > > > > > > > > > applicable.
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > I agree that the format is intended to be general
> > > > purpose,
> > > > > > but
> > > > > > > > > types
> > > > > > > > > > > > within
> > > > > > > > > > > > > the format define specific representation and
> > semantics
> > > > to
> > > > > > > > achieve
> > > > > > > > > > that
> > > > > > > > > > > > > goal.
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > > "Vector" has a more general sense than ML and
> it's
> > > > quite
> > > > > > > > trivial
> > > > > > > > > to
> > > > > > > > > > > > > > produce vectors with non-finite values using e.g.
> > > NumPy
> > > > > or
> > > > > > > > Pandas
> > > > > > > > > > (or
> > > > > > > > > > > > > > any other piece of dataframe or scientific
> > computing
> > > > > > > software).
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >  The justification for restricting non-finite
> values
> > is
> > > > > > > directly
> > > > > > > > > > > related
> > > > > > > > > > > > to
> > > > > > > > > > > > > the semantic meaning of a vector.  Vectors have
> > > specific
> > > > > > > > > > > characteristics
> > > > > > > > > > > > > (dimension, magnitude, direction, etc).  Non-finite
> > > > values
> > > > > > are
> > > > > > > > > > invalid
> > > > > > > > > > > > and
> > > > > > > > > > > > > break those characteristics.  I'd liken it to
> storing
> > > > > > non-utf8
> > > > > > > > > values
> > > > > > > > > > > in
> > > > > > > > > > > > a
> > > > > > > > > > > > > string and still calling it a string.
> > > > > > > > > > > > >
> > > > > > > > > > > > > That also differentiates a vector from a fixed
> length
> > > > list,
> > > > > > > which
> > > > > > > > > is
> > > > > > > > > > > more
> > > > > > > > > > > > > general and does not carry the same semantic
> meaning.
> > > > > > > > > > > > >
> > > > > > > > > > > > > I also don't think we need to bundle all similarly
> > > shaped
> > > > > use
> > > > > > > > cases
> > > > > > > > > > > into
> > > > > > > > > > > > > the same solution (e.g. flxed size list).  Vector
> is
> > a
> > > > well
> > > > > > > > > > established
> > > > > > > > > > > > > data type at this point and aligns with the general
> > > > purpose
> > > > > > > > nature
> > > > > > > > > of
> > > > > > > > > > > the
> > > > > > > > > > > > > format (many other formats and databases natively
> > > support
> > > > > > > this).
> > > > > > > > > > > > >
> > > > > > > > > > > > > -Dan
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Thu, Sep 24, 2026 at 11:55 AM Antoine Pitrou <
> > > > > > > > > antoine@python.org>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Le 24/09/2026 à 20:11, Russell Spitzer a écrit :
> > > > > > > > > > > > > > > I think something I haven't seen yet is the
> clear
> > > > > > use-case
> > > > > > > > for
> > > > > > > > > > > > > non-finite
> > > > > > > > > > > > > > > values within a vector type. There has been a
> lot
> > > of
> > > > > > > > > theoretical
> > > > > > > > > > > > > > discussion
> > > > > > > > > > > > > > > here about use cases but what toolings
> currently
> > > > > produce
> > > > > > > > these
> > > > > > > > > > > > vectors
> > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > how do they function in downstream engines? I'd
> > > like
> > > > to
> > > > > > > limit
> > > > > > > > > > > > > introducing
> > > > > > > > > > > > > > > types that have a wide range of values which
> are
> > > > simply
> > > > > > are
> > > > > > > > > > failure
> > > > > > > > > > > > > modes
> > > > > > > > > > > > > > > of different flavors
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > "Vector" has a more general sense than ML and
> it's
> > > > quite
> > > > > > > > trivial
> > > > > > > > > to
> > > > > > > > > > > > > > produce vectors with non-finite values using e.g.
> > > NumPy
> > > > > or
> > > > > > > > Pandas
> > > > > > > > > > (or
> > > > > > > > > > > > > > any other piece of dataframe or scientific
> > computing
> > > > > > > software).
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > It's not like the general concept of a vector is
> a
> > > IETF
> > > > > > > > standard
> > > > > > > > > or
> > > > > > > > > > > > > > anything, so analogies with JSON or UTF8 (where
> the
> > > > > values
> > > > > > > are
> > > > > > > > > > > expected
> > > > > > > > > > > > > > to be valid according to the respective specs) do
> > not
> > > > > > apply.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > This discussion would IMHO have a better chance
> of
> > > > > > converging
> > > > > > > > if
> > > > > > > > > we
> > > > > > > > > > > > > > separated the two orthogonal concerns of:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 1. representing fixed-length list/vector data in
> > > > Parquet
> > > > > > > (with
> > > > > > > > a
> > > > > > > > > > > > > > potential limitation to non-nested types)
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 2. representing the information that a given
> column
> > > (or
> > > > > > > column
> > > > > > > > > > chunk,
> > > > > > > > > > > > > > etc.) does not contain non-finite values
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > ... both of which are useful on their own right,
> > but
> > > do
> > > > > not
> > > > > > > > have
> > > > > > > > > to
> > > > > > > > > > > be
> > > > > > > > > > > > > > conflated and bundled together.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Regards
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Antoine.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] Numeric Vector type: physical representation (outcome of focused call)

[Alkis Evlogimenos via dev <de...@parquet.apache.org>]

I suggest we go with VECTOR for the finite vector to keep the name the same
as other engines/type systems. If we want a non-finite one in the future we
can name that something else or we add fixed size list that will serve the
same purpose without a special name.


On Thu, Oct 8, 2026 at 10:20 PM Ryan Blue <bl...@apache.org> wrote:

> A logical type would:
> - Describe data for users and readers to understand what the file contains
> - Signal that repetition and definition levels will reflect a fixed number
> of elements, which implementations can use to optimize decoding
> - Guarantee that there are no null elements (because the type cannot be
> applied to a list with optional elements)
> - [FINITE_VECTOR] Guarantee that no elements are NaN or Infinity values so
> that the vectors have a finite length and are suitable for distance
> calculations
>
> The finite value restriction is a restriction on writers to prevent bad
> data from being written.
>
> Ignoring logical types that are not understood is a rule for readers. A
> reader can handle vectors and pass it on to another layer. For instance, a
> Parquet reader that doesn't support geometry passes values as `binary`.
> That binary value can be passed to a geometry UDF, even if the underlying
> engine and Parquet reader doesn't have support.
>
> A tool that doesn't understand FINITE_VECTOR can similarly pass a list of
> floats along to a UDF. But it is still important to the user that bad
> values are not present in the data. In your example, bad values are vectors
> that cause the centroid calculation to fail, like vectors with infinite
> length.
>
> Ryan
>
> On Thu, Oct 8, 2026 at 12:54 PM Will Edwards via dev <
> dev@parquet.apache.org>
> wrote:
>
> > Could we clarify the use cases for a VECTOR annotation on LIST<FLOAT>?
> > Specifically, what does it make possible that's impossible today, and
> what
> > does it make easier that's currently hard?
> >
> > I'm struggling to find a strong answer. I'd like every engine to offer
> > vector functions on lists of floats without needing a new logical type.
> >
> > To be clear, I'm still enthusiastic about a vector type that describes
> how
> > to unpack and scale composite representations (quantised codes plus
> scales,
> > codebooks, etc.), in the direction ML is going. That adds information
> > engines don't have. A label on LIST<FLOAT> doesn't: engines already
> > understand the physical type, and many already offer vector functions on
> > it. I don't see a performance benefit, and NaN/null/Inf checks are cheap
> > enough that engines should do them anyway. The label also doesn't say
> what
> > you can do with the column. Knowing that a column holds embeddings
> doesn't
> > say how they can be used, just as zipcode and age are both integers but
> > multiplying them is meaningless.
> >
> > My concern is the opposite effect. We're establishing that engines should
> > ignore logical types they don't understand. So a tool that doesn't know
> > VECTOR will read the column as LIST<FLOAT>, let the user compute
> centroids
> > (say), and write plain LIST<FLOAT> back out. If other engines restrict
> > embedding functions to VECTOR-annotated columns, that file stops working
> > with them, through no fault of the user. An annotation that doesn't
> change
> > what the data is shouldn't be able to make it unusable.
> >
> > Am I missing a use case?
> >
> > Best, Will
> >
> > On Thu, 8 Oct 2026 at 19:53, Steven Wu <st...@gmail.com> wrote:
> >
> > > I also think a FINITE_VECTOR type brings a lot of values to many use
> > cases.
> > > It is would be great to make progress on the FINITE_VECTOR, and not to
> > fit
> > > every use cases into the FIXED_LENGTH_LIST.
> > >
> > > On Mon, Oct 5, 2026 at 1:26 PM Ryan Blue <bl...@apache.org> wrote:
> > >
> > > > Going through this thread again, it looks like there are a few things
> > > that
> > > > people generally agree on:
> > > >
> > > > - In the short term, we want to introduce a logical type or types; a
> > > better
> > > > physical representation can be done later. We can focus on logical
> > types.
> > > > - There is precedent in logical types for restricting values, like
> UTF8
> > > > strings to contain only UTF-8 codepoints. I think we also agree that
> we
> > > > could restrict vector elements to non-nested or primitive types.
> > > > - There are many examples of databases that restrict their vector
> types
> > > to
> > > > non-null, finite elements.
> > > > - Some people want to have write-time guarantees that vectors have
> only
> > > > non-null, finite elements
> > > > - Some people want to extend this work to include tensors, shorter
> > > floating
> > > > point representations, variable lengths, NaN or Infinity elements,
> etc.
> > > >
> > > > It sounds like we are nearly there, but have two halves of the
> > community
> > > > that care about different things. That just means we should have
> > multiple
> > > > types. It's entirely reasonable to want a type that allows NaN
> values,
> > > and
> > > > it's also reasonable for those of us building databases, table
> > standards,
> > > > and vector indexes to care about guarantees that elements are finite.
> > > > There's more than enough examples to show that this is a legitimate
> use
> > > > case.
> > > >
> > > > Since I don't see the concerns of the different use cases changing,
> it
> > > > looks like we need to find a compromise to move forward. I propose
> that
> > > we
> > > > introduce a FINITE_VECTOR type and separately a FIXED_LENGTH_LIST or
> > > > another VECTOR type that allows infinite and NaN values.
> > > >
> > > > Can we agree on a compromise? Is there a strong reason to try to fit
> > > these
> > > > use cases into a single type? What if that means not making progress?
> > > >
> > > > Ryan
> > > >
> > > > On Mon, Oct 5, 2026 at 12:15 PM Philipp Fischbeck via dev <
> > > > dev@parquet.apache.org> wrote:
> > > >
> > > > > Thinking about this more, I am also fine with pulling out the
> > “NaN/Inf”
> > > > > requirement. Either into a type boolean (my preference), or putting
> > no
> > > > > restrictions on the Parquet level, allowing all IEEE 754 float
> > numbers.
> > > > If
> > > > > some top-level application wants to put additional constraints on
> the
> > > > data
> > > > > type, like Iceberg not allowing NaN/infinity, it can still do so
> one
> > > > layer
> > > > > above. It seems like this was the biggest point of discussion
> > > remaining.
> > > > > All these options work for me, and we should move forward with this
> > > > > proposal.
> > > > >
> > > > > Concerning additional element types like bf16, they can be
> introduced
> > > as
> > > > > Parquet types (at least in the current LIST-based spec proposal).
> > > > >
> > > > > Concerning Will’s suggestion around representing VECTOR as a
> variable
> > > > > length list with a MIN/MAX_LENGTH metadata restriction: I think
> that
> > a
> > > > > dedicated logical VECTOR type still provides enough benefits by
> > > > describing
> > > > > the semantics of the data, and allowing to capture future changes
> > > around
> > > > > statistics. In that case, the length is better captured as an exact
> > > > length
> > > > > for this type specifically. I’m also not sure about other use cases
> > for
> > > > > MIN_LENGTH, so introducing a new type constraint system just for
> the
> > > > > MIN_LENGTH=MAX_LENGTH case for vectors doesn’t have enough value in
> > my
> > > > > view.
> > > > >
> > > > > Best,
> > > > > Philipp
> > > > >
> > > > >
> > > > > On Thu, Oct 1, 2026 at 9:00 AM Will Edwards via dev <
> > > > > dev@parquet.apache.org>
> > > > > wrote:
> > > > >
> > > > > > +1 Sebastian, a vector type suggests usage beyond normalized
> > > embeddings
> > > > > and
> > > > > > engines really aren't going to be able to elide all the checks
> > > anyway.
> > > > > >
> > > > > > Antoine, you suggested we explore metadata solutions too :)
> > > > > >
> > > > > > Today, a database supporting CHAR(N) or VARCHAR(n) can't express
> > this
> > > > in
> > > > > > Parquet.  Engines can write opaque blobs they understand in the
> > > > metadata
> > > > > > (e.g. Arrow does) so the extra type info round-trips, but this
> > isn't
> > > > > > standardized, portable, nor interoperable.
> > > > > >
> > > > > > So perhaps standardizing constraint metadata or new Thrift fields
> > (my
> > > > > pref)
> > > > > > can generically support VECTOR(N) whilst also being more broadly
> > > useful
> > > > > for
> > > > > > other use cases and data types?
> > > > > >
> > > > > > Imagine metadata or Thrift fields to tag the MIN_LENGTH and
> > > MAX_LENGTH
> > > > > for
> > > > > > Parquet's variable-length physcial types.  These would count
> > logical
> > > > > units
> > > > > > (e.g. UTF8 code points for STRING) rather than bytes.
> > > > > >
> > > > > > Then one engine's VECTOR(N) can be written to Parquet as LIST(N)
> > with
> > > > > > MIN_LENGTH = MAX_LENGTH = N and then readers - even different
> > > engines -
> > > > > can
> > > > > > pick up these constraints if they support doing so.  We don't
> > > actually
> > > > > need
> > > > > > a logical type to express a VECTOR then.
> > > > > >
> > > > > > This also allows standard SQL's CHAR(N) and VARCHAR(N) to be
> > > expressed
> > > > on
> > > > > > UTF8 columns etc.
> > > > > >
> > > > > > This also seems generally useful in the future as we approach
> > > > expressing
> > > > > > other tensor data shapes and types.
> > > > > >
> > > > > > This won't change the mechanics: the storage type doesn't enforce
> > > these
> > > > > > constraints and engines must check them. It's previously
> > established
> > > > that
> > > > > > performance is just a software problem and there is no
> fundamental
> > > > > > performance advantage to the fixed size types in Dremel-like
> > formats
> > > > like
> > > > > > Parquet.
> > > > > >
> > > > > > Best,
> > > > > > Will
> > > > > >
> > > > > > On Wed, 30 Sept 2026 at 16:08, Sebastian Baunsgaard <
> > > > > baunsgaard@apache.org
> > > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > After listening in on the last meeting Wednesday, I have been
> > > > thinking
> > > > > > > about this quite a bit, and thought I would share my 2 cents.
> > > > > > >
> > > > > > > I suggest not requiring VECTOR elements to be finite, as
> proposed
> > > in
> > > > > the
> > > > > > PR
> > > > > > > <https://github.com/apache/parquet-format/pull/624>.
> > > > > > >
> > > > > > > While Divjot and others note the specialization for UTF-8,
> > Parquet
> > > > > > already
> > > > > > > defines DOUBLE and FLOAT as IEEE encodings
> > > > > > > <
> > > > https://github.com/apache/parquet-format/blob/master/README.md#types
> > > > > >.
> > > > > > >
> > > > > > > Making NaN or +/-Inf illegal inside VECTOR would be a new
> > > restriction
> > > > > on
> > > > > > > those same underlying physical types. Scalar FLOAT/DOUBLE
> columns
> > > > have
> > > > > > > never had such a rule.
> > > > > > >
> > > > > > > It would also not match how the systems that actually produce
> > these
> > > > > > vectors
> > > > > > > treat floats. Most of them use IEEE floats, e.g. Spark, DuckDB,
> > > > NumPy,
> > > > > > > PyTorch, TensorFlow, or lower level CUDA, oneMKL, and OpenBLAS.
> > > > > > >
> > > > > > > pgvector's "all elements must be finite" rule is an insert
> > policy.
> > > > > While
> > > > > > I
> > > > > > > think Parquet as a general-purpose format should allow them out
> > of
> > > > the
> > > > > > box.
> > > > > > > It is a user system decision to disallow writing these values
> or
> > to
> > > > > throw
> > > > > > > errors when encountering them.
> > > > > > >
> > > > > > > A type-level ban also does not keep Inf/NaN out of vector math.
> > > > Finite
> > > > > > > coordinates still overflow. For instance: magnitude =
> > ||[FLOAT_MAX,
> > > > > > > FLOAT_MAX]|| where every coordinate is finite overflows to
> +Inf.
> > > > > PyTorch
> > > > > > > documents the same overflow: tensor([1e20, 1e20]).norm() is inf
> > in
> > > > > fp32.
> > > > > > >
> > > > > > > A generated MAGNITUDE(vec) column would then be Inf while the
> > > VECTOR
> > > > > > itself
> > > > > > > was required to be finite. The ban only forbids storing
> specials
> > in
> > > > the
> > > > > > > coordinates, and does not prevent even simple downstream
> > > computations
> > > > > > from
> > > > > > > producing them.
> > > > > > >
> > > > > > > I suggest keeping VECTOR as the type and add an optional bool
> > > > > > > VectorType.finite (default false). Engines that want
> > pgvector-like
> > > > > > > rejection can require the flag = true. Engines that use NumPy /
> > > > > PyTorch /
> > > > > > > Spark arrays can keep representable IEEE values. For the same
> > > > reason, I
> > > > > > do
> > > > > > > not favor adding yet another type called FINITE_VECTOR.
> > > > > > >
> > > > > > > Best,
> > > > > > > Sebastian
> > > > > > >
> > > > > > >
> > > > > > > On Tue, Sep 29, 2026 at 2:03 PM Divjot Arora via dev <
> > > > > > > dev@parquet.apache.org>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > I agree with moving forward with the proposal and renaming
> > VECTOR
> > > > to
> > > > > > > > FINITE_VECTOR or something
> > > > > > > > similar if others feel that better communicates its intent. I
> > see
> > > > > value
> > > > > > > in
> > > > > > > > having a vector type that guarantees
> > > > > > > > its values are mathematically-correct vectors for a few
> > reasons:
> > > > > > > >
> > > > > > > > 1. As others have mentioned, this is similar to the STRING
> > > logical
> > > > > type
> > > > > > > > mandating UTF-8.
> > > > > > > > 2. It is not possible to rely on statistics for detecting NaN
> > or
> > > > > +-inf
> > > > > > > > values as all statistics are optional.
> > > > > > > > To my knowledge, there is no precedent for requiring that
> > writers
> > > > > emit
> > > > > > > > stats for specific logical types.
> > > > > > > > 3. Without statistics, NaN/inf values could flow through the
> > > engine
> > > > > and
> > > > > > > > block them for specific functionality.
> > > > > > > > This seems to be Lance's behavior. However, I don't think
> this
> > is
> > > > as
> > > > > > easy
> > > > > > > > as it sounds and Lance has run
> > > > > > > > into several bugs due to this ([1] [2] for examples). Their
> > > Python
> > > > > [3]
> > > > > > > and
> > > > > > > > Rust [4] APIs default to erroring on
> > > > > > > > vectors that contain NaN values.
> > > > > > > >
> > > > > > > > I agree that Parquet is a general-purpose format, but I don't
> > > think
> > > > > > this
> > > > > > > > implies the lack of domain-specific types.
> > > > > > > > Instead, this broadens the use cases where Parquet would be
> > > > > considered
> > > > > > a
> > > > > > > > good fit. I also fully agree with
> > > > > > > > continuing work on a general fixed-size list type if we
> believe
> > > > there
> > > > > > are
> > > > > > > > non-vector use cases that would benefit
> > > > > > > > from such guarantees at the storage layer.
> > > > > > > >
> > > > > > > > [1] https://github.com/lance-format/lance/pull/3648
> > > > > > > > [2] https://github.com/lance-format/lance/issues/8607 (see
> > > point 2
> > > > > in
> > > > > > > > "root
> > > > > > > > cause")
> > > > > > > > [3]
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/lancedb/lancedb/blob/0327618a62483feb6dca2336549845e5887cd48f/python/python/lancedb/table.py#L4807-L4911
> > > > > > > > [4]
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/lancedb/lancedb/blob/0327618a62483feb6dca2336549845e5887cd48f/rust/lancedb/src/table/add_data.rs#L105-L110
> > > > > > > >
> > > > > > > > On Mon, Sep 28, 2026 at 6:19 PM Philipp Fischbeck via dev <
> > > > > > > > dev@parquet.apache.org> wrote:
> > > > > > > >
> > > > > > > > > I also favor moving forward with the proposal as is. I see
> > > strong
> > > > > > value
> > > > > > > > and
> > > > > > > > > use cases for the numeric vector type, and I agree with the
> > > > > > > restrictions
> > > > > > > > > around NaN/Inf values and element types in general.
> Precedent
> > > > > exists
> > > > > > in
> > > > > > > > > other types that restrict which physical type values are
> > > > logically
> > > > > > > valid
> > > > > > > > > (e.g., the geospatial types needing to be WKB-encoded).  A
> > > > general
> > > > > > > > > fixed-size list also has value, but I think it makes sense
> to
> > > > > > separate
> > > > > > > > the
> > > > > > > > > two types. The semantics are relevant, and I can see how
> the
> > > two
> > > > > > types
> > > > > > > > > might diverge further in the future regarding statistics or
> > > > > > encodings.
> > > > > > > > >
> > > > > > > > > I would be okay with renaming the vector type, but it seems
> > > like
> > > > > the
> > > > > > > > VECTOR
> > > > > > > > > name is already used across other engines mentioned in this
> > > > thread.
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > > Philipp
> > > > > > > > >
> > > > > > > > > On Sat, Sep 26, 2026 at 4:37 PM Gang Wu <us...@gmail.com>
> > > > wrote:
> > > > > > > > >
> > > > > > > > > > To separate orthogonal concerns, I feel we should treat
> the
> > > > > general
> > > > > > > > > > FixedSizeList as the underlying physical layout, and
> > separate
> > > > it
> > > > > > from
> > > > > > > > the
> > > > > > > > > > discussion of a specialized VECTOR (or broader ML tensor)
> > > > logical
> > > > > > > type.
> > > > > > > > > >
> > > > > > > > > > Currently, the immediate need is to define a specialized
> > > VECTOR
> > > > > > type
> > > > > > > so
> > > > > > > > > > that different engines and formats have a common semantic
> > > > > contract
> > > > > > > for
> > > > > > > > > > interoperability.
> > > > > > > > > >
> > > > > > > > > > If the main point of contention is just the NaN/Inf/null
> > > > > > limitations,
> > > > > > > > > maybe
> > > > > > > > > > we can resolve this by adding a qualification to the name
> > > (like
> > > > > > > Andrew
> > > > > > > > > > suggested, e.g., VECTOR_EMBEDDING or FINITE_VECTOR) to
> > avoid
> > > > > > > confusion
> > > > > > > > > with
> > > > > > > > > > broader ML use cases. Alternatively, we could introduce a
> > > > boolean
> > > > > > > flag
> > > > > > > > > > (e.g., optional bool VectorType.finite) as proposed
> > earlier.
> > > > > > > > > >
> > > > > > > > > > On Fri, Sep 25, 2026 at 3:52 AM Andrew Lamb <
> > > > > > andrewlamb11@gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > I am a little confused about the points of disagreement
> > > here.
> > > > > > > > > > >
> > > > > > > > > > > Is it that the name `VECTOR` is too general for the
> > > specific
> > > > > use
> > > > > > > case
> > > > > > > > > of
> > > > > > > > > > > fixed length arrays with non-finite values used for
> > vector
> > > > > > > > embeddings?
> > > > > > > > > > If
> > > > > > > > > > > so, maybe we could add a qualification to the name
> > > > > > (FINITE_VECTOR?
> > > > > > > > > > > FEATURE_VECTOR? VECTOR_EMBEDDING?)
> > > > > > > > > > >
> > > > > > > > > > > It seems like there is precedent for defining logical
> > types
> > > > > that
> > > > > > > > > specify
> > > > > > > > > > > certain physical types and restrict their contents to
> > help
> > > > > > readers
> > > > > > > > > > > interpret meaning, for example string[1] or JSON;
> > Defining
> > > > one
> > > > > > for
> > > > > > > > > > vector
> > > > > > > > > > > embeddings seems reasonable enough to me.
> > > > > > > > > > >
> > > > > > > > > > > Andrew
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > [1]:
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#string
> > > > > > > > > > >
> > > > > > > > > > > On Thu, Sep 24, 2026 at 3:34 PM Daniel Weeks <
> > > > > dweeks@apache.org>
> > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Well, Parquet is not a single-purpose format.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Vector stores can choose whatever policy fits their
> > > > usage,
> > > > > > but
> > > > > > > > > we're
> > > > > > > > > > > > > talking about a general-purpose file format that
> aims
> > > to
> > > > be
> > > > > > > > broadly
> > > > > > > > > > > > > applicable.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > I agree that the format is intended to be general
> > > purpose,
> > > > > but
> > > > > > > > types
> > > > > > > > > > > within
> > > > > > > > > > > > the format define specific representation and
> semantics
> > > to
> > > > > > > achieve
> > > > > > > > > that
> > > > > > > > > > > > goal.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > > "Vector" has a more general sense than ML and it's
> > > quite
> > > > > > > trivial
> > > > > > > > to
> > > > > > > > > > > > > produce vectors with non-finite values using e.g.
> > NumPy
> > > > or
> > > > > > > Pandas
> > > > > > > > > (or
> > > > > > > > > > > > > any other piece of dataframe or scientific
> computing
> > > > > > software).
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >  The justification for restricting non-finite values
> is
> > > > > > directly
> > > > > > > > > > related
> > > > > > > > > > > to
> > > > > > > > > > > > the semantic meaning of a vector.  Vectors have
> > specific
> > > > > > > > > > characteristics
> > > > > > > > > > > > (dimension, magnitude, direction, etc).  Non-finite
> > > values
> > > > > are
> > > > > > > > > invalid
> > > > > > > > > > > and
> > > > > > > > > > > > break those characteristics.  I'd liken it to storing
> > > > > non-utf8
> > > > > > > > values
> > > > > > > > > > in
> > > > > > > > > > > a
> > > > > > > > > > > > string and still calling it a string.
> > > > > > > > > > > >
> > > > > > > > > > > > That also differentiates a vector from a fixed length
> > > list,
> > > > > > which
> > > > > > > > is
> > > > > > > > > > more
> > > > > > > > > > > > general and does not carry the same semantic meaning.
> > > > > > > > > > > >
> > > > > > > > > > > > I also don't think we need to bundle all similarly
> > shaped
> > > > use
> > > > > > > cases
> > > > > > > > > > into
> > > > > > > > > > > > the same solution (e.g. flxed size list).  Vector is
> a
> > > well
> > > > > > > > > established
> > > > > > > > > > > > data type at this point and aligns with the general
> > > purpose
> > > > > > > nature
> > > > > > > > of
> > > > > > > > > > the
> > > > > > > > > > > > format (many other formats and databases natively
> > support
> > > > > > this).
> > > > > > > > > > > >
> > > > > > > > > > > > -Dan
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > On Thu, Sep 24, 2026 at 11:55 AM Antoine Pitrou <
> > > > > > > > antoine@python.org>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Le 24/09/2026 à 20:11, Russell Spitzer a écrit :
> > > > > > > > > > > > > > I think something I haven't seen yet is the clear
> > > > > use-case
> > > > > > > for
> > > > > > > > > > > > non-finite
> > > > > > > > > > > > > > values within a vector type. There has been a lot
> > of
> > > > > > > > theoretical
> > > > > > > > > > > > > discussion
> > > > > > > > > > > > > > here about use cases but what toolings currently
> > > > produce
> > > > > > > these
> > > > > > > > > > > vectors
> > > > > > > > > > > > > and
> > > > > > > > > > > > > > how do they function in downstream engines? I'd
> > like
> > > to
> > > > > > limit
> > > > > > > > > > > > introducing
> > > > > > > > > > > > > > types that have a wide range of values which are
> > > simply
> > > > > are
> > > > > > > > > failure
> > > > > > > > > > > > modes
> > > > > > > > > > > > > > of different flavors
> > > > > > > > > > > > >
> > > > > > > > > > > > > "Vector" has a more general sense than ML and it's
> > > quite
> > > > > > > trivial
> > > > > > > > to
> > > > > > > > > > > > > produce vectors with non-finite values using e.g.
> > NumPy
> > > > or
> > > > > > > Pandas
> > > > > > > > > (or
> > > > > > > > > > > > > any other piece of dataframe or scientific
> computing
> > > > > > software).
> > > > > > > > > > > > >
> > > > > > > > > > > > > It's not like the general concept of a vector is a
> > IETF
> > > > > > > standard
> > > > > > > > or
> > > > > > > > > > > > > anything, so analogies with JSON or UTF8 (where the
> > > > values
> > > > > > are
> > > > > > > > > > expected
> > > > > > > > > > > > > to be valid according to the respective specs) do
> not
> > > > > apply.
> > > > > > > > > > > > >
> > > > > > > > > > > > > This discussion would IMHO have a better chance of
> > > > > converging
> > > > > > > if
> > > > > > > > we
> > > > > > > > > > > > > separated the two orthogonal concerns of:
> > > > > > > > > > > > >
> > > > > > > > > > > > > 1. representing fixed-length list/vector data in
> > > Parquet
> > > > > > (with
> > > > > > > a
> > > > > > > > > > > > > potential limitation to non-nested types)
> > > > > > > > > > > > >
> > > > > > > > > > > > > 2. representing the information that a given column
> > (or
> > > > > > column
> > > > > > > > > chunk,
> > > > > > > > > > > > > etc.) does not contain non-finite values
> > > > > > > > > > > > >
> > > > > > > > > > > > > ... both of which are useful on their own right,
> but
> > do
> > > > not
> > > > > > > have
> > > > > > > > to
> > > > > > > > > > be
> > > > > > > > > > > > > conflated and bundled together.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Regards
> > > > > > > > > > > > >
> > > > > > > > > > > > > Antoine.
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] Numeric Vector type: physical representation (outcome of focused call)

[Ryan Blue <bl...@apache.org>]

A logical type would:
- Describe data for users and readers to understand what the file contains
- Signal that repetition and definition levels will reflect a fixed number
of elements, which implementations can use to optimize decoding
- Guarantee that there are no null elements (because the type cannot be
applied to a list with optional elements)
- [FINITE_VECTOR] Guarantee that no elements are NaN or Infinity values so
that the vectors have a finite length and are suitable for distance
calculations

The finite value restriction is a restriction on writers to prevent bad
data from being written.

Ignoring logical types that are not understood is a rule for readers. A
reader can handle vectors and pass it on to another layer. For instance, a
Parquet reader that doesn't support geometry passes values as `binary`.
That binary value can be passed to a geometry UDF, even if the underlying
engine and Parquet reader doesn't have support.

A tool that doesn't understand FINITE_VECTOR can similarly pass a list of
floats along to a UDF. But it is still important to the user that bad
values are not present in the data. In your example, bad values are vectors
that cause the centroid calculation to fail, like vectors with infinite
length.

Ryan

On Thu, Oct 8, 2026 at 12:54 PM Will Edwards via dev <de...@parquet.apache.org>
wrote:

> Could we clarify the use cases for a VECTOR annotation on LIST<FLOAT>?
> Specifically, what does it make possible that's impossible today, and what
> does it make easier that's currently hard?
>
> I'm struggling to find a strong answer. I'd like every engine to offer
> vector functions on lists of floats without needing a new logical type.
>
> To be clear, I'm still enthusiastic about a vector type that describes how
> to unpack and scale composite representations (quantised codes plus scales,
> codebooks, etc.), in the direction ML is going. That adds information
> engines don't have. A label on LIST<FLOAT> doesn't: engines already
> understand the physical type, and many already offer vector functions on
> it. I don't see a performance benefit, and NaN/null/Inf checks are cheap
> enough that engines should do them anyway. The label also doesn't say what
> you can do with the column. Knowing that a column holds embeddings doesn't
> say how they can be used, just as zipcode and age are both integers but
> multiplying them is meaningless.
>
> My concern is the opposite effect. We're establishing that engines should
> ignore logical types they don't understand. So a tool that doesn't know
> VECTOR will read the column as LIST<FLOAT>, let the user compute centroids
> (say), and write plain LIST<FLOAT> back out. If other engines restrict
> embedding functions to VECTOR-annotated columns, that file stops working
> with them, through no fault of the user. An annotation that doesn't change
> what the data is shouldn't be able to make it unusable.
>
> Am I missing a use case?
>
> Best, Will
>
> On Thu, 8 Oct 2026 at 19:53, Steven Wu <st...@gmail.com> wrote:
>
> > I also think a FINITE_VECTOR type brings a lot of values to many use
> cases.
> > It is would be great to make progress on the FINITE_VECTOR, and not to
> fit
> > every use cases into the FIXED_LENGTH_LIST.
> >
> > On Mon, Oct 5, 2026 at 1:26 PM Ryan Blue <bl...@apache.org> wrote:
> >
> > > Going through this thread again, it looks like there are a few things
> > that
> > > people generally agree on:
> > >
> > > - In the short term, we want to introduce a logical type or types; a
> > better
> > > physical representation can be done later. We can focus on logical
> types.
> > > - There is precedent in logical types for restricting values, like UTF8
> > > strings to contain only UTF-8 codepoints. I think we also agree that we
> > > could restrict vector elements to non-nested or primitive types.
> > > - There are many examples of databases that restrict their vector types
> > to
> > > non-null, finite elements.
> > > - Some people want to have write-time guarantees that vectors have only
> > > non-null, finite elements
> > > - Some people want to extend this work to include tensors, shorter
> > floating
> > > point representations, variable lengths, NaN or Infinity elements, etc.
> > >
> > > It sounds like we are nearly there, but have two halves of the
> community
> > > that care about different things. That just means we should have
> multiple
> > > types. It's entirely reasonable to want a type that allows NaN values,
> > and
> > > it's also reasonable for those of us building databases, table
> standards,
> > > and vector indexes to care about guarantees that elements are finite.
> > > There's more than enough examples to show that this is a legitimate use
> > > case.
> > >
> > > Since I don't see the concerns of the different use cases changing, it
> > > looks like we need to find a compromise to move forward. I propose that
> > we
> > > introduce a FINITE_VECTOR type and separately a FIXED_LENGTH_LIST or
> > > another VECTOR type that allows infinite and NaN values.
> > >
> > > Can we agree on a compromise? Is there a strong reason to try to fit
> > these
> > > use cases into a single type? What if that means not making progress?
> > >
> > > Ryan
> > >
> > > On Mon, Oct 5, 2026 at 12:15 PM Philipp Fischbeck via dev <
> > > dev@parquet.apache.org> wrote:
> > >
> > > > Thinking about this more, I am also fine with pulling out the
> “NaN/Inf”
> > > > requirement. Either into a type boolean (my preference), or putting
> no
> > > > restrictions on the Parquet level, allowing all IEEE 754 float
> numbers.
> > > If
> > > > some top-level application wants to put additional constraints on the
> > > data
> > > > type, like Iceberg not allowing NaN/infinity, it can still do so one
> > > layer
> > > > above. It seems like this was the biggest point of discussion
> > remaining.
> > > > All these options work for me, and we should move forward with this
> > > > proposal.
> > > >
> > > > Concerning additional element types like bf16, they can be introduced
> > as
> > > > Parquet types (at least in the current LIST-based spec proposal).
> > > >
> > > > Concerning Will’s suggestion around representing VECTOR as a variable
> > > > length list with a MIN/MAX_LENGTH metadata restriction: I think that
> a
> > > > dedicated logical VECTOR type still provides enough benefits by
> > > describing
> > > > the semantics of the data, and allowing to capture future changes
> > around
> > > > statistics. In that case, the length is better captured as an exact
> > > length
> > > > for this type specifically. I’m also not sure about other use cases
> for
> > > > MIN_LENGTH, so introducing a new type constraint system just for the
> > > > MIN_LENGTH=MAX_LENGTH case for vectors doesn’t have enough value in
> my
> > > > view.
> > > >
> > > > Best,
> > > > Philipp
> > > >
> > > >
> > > > On Thu, Oct 1, 2026 at 9:00 AM Will Edwards via dev <
> > > > dev@parquet.apache.org>
> > > > wrote:
> > > >
> > > > > +1 Sebastian, a vector type suggests usage beyond normalized
> > embeddings
> > > > and
> > > > > engines really aren't going to be able to elide all the checks
> > anyway.
> > > > >
> > > > > Antoine, you suggested we explore metadata solutions too :)
> > > > >
> > > > > Today, a database supporting CHAR(N) or VARCHAR(n) can't express
> this
> > > in
> > > > > Parquet.  Engines can write opaque blobs they understand in the
> > > metadata
> > > > > (e.g. Arrow does) so the extra type info round-trips, but this
> isn't
> > > > > standardized, portable, nor interoperable.
> > > > >
> > > > > So perhaps standardizing constraint metadata or new Thrift fields
> (my
> > > > pref)
> > > > > can generically support VECTOR(N) whilst also being more broadly
> > useful
> > > > for
> > > > > other use cases and data types?
> > > > >
> > > > > Imagine metadata or Thrift fields to tag the MIN_LENGTH and
> > MAX_LENGTH
> > > > for
> > > > > Parquet's variable-length physcial types.  These would count
> logical
> > > > units
> > > > > (e.g. UTF8 code points for STRING) rather than bytes.
> > > > >
> > > > > Then one engine's VECTOR(N) can be written to Parquet as LIST(N)
> with
> > > > > MIN_LENGTH = MAX_LENGTH = N and then readers - even different
> > engines -
> > > > can
> > > > > pick up these constraints if they support doing so.  We don't
> > actually
> > > > need
> > > > > a logical type to express a VECTOR then.
> > > > >
> > > > > This also allows standard SQL's CHAR(N) and VARCHAR(N) to be
> > expressed
> > > on
> > > > > UTF8 columns etc.
> > > > >
> > > > > This also seems generally useful in the future as we approach
> > > expressing
> > > > > other tensor data shapes and types.
> > > > >
> > > > > This won't change the mechanics: the storage type doesn't enforce
> > these
> > > > > constraints and engines must check them. It's previously
> established
> > > that
> > > > > performance is just a software problem and there is no fundamental
> > > > > performance advantage to the fixed size types in Dremel-like
> formats
> > > like
> > > > > Parquet.
> > > > >
> > > > > Best,
> > > > > Will
> > > > >
> > > > > On Wed, 30 Sept 2026 at 16:08, Sebastian Baunsgaard <
> > > > baunsgaard@apache.org
> > > > > >
> > > > > wrote:
> > > > >
> > > > > > After listening in on the last meeting Wednesday, I have been
> > > thinking
> > > > > > about this quite a bit, and thought I would share my 2 cents.
> > > > > >
> > > > > > I suggest not requiring VECTOR elements to be finite, as proposed
> > in
> > > > the
> > > > > PR
> > > > > > <https://github.com/apache/parquet-format/pull/624>.
> > > > > >
> > > > > > While Divjot and others note the specialization for UTF-8,
> Parquet
> > > > > already
> > > > > > defines DOUBLE and FLOAT as IEEE encodings
> > > > > > <
> > > https://github.com/apache/parquet-format/blob/master/README.md#types
> > > > >.
> > > > > >
> > > > > > Making NaN or +/-Inf illegal inside VECTOR would be a new
> > restriction
> > > > on
> > > > > > those same underlying physical types. Scalar FLOAT/DOUBLE columns
> > > have
> > > > > > never had such a rule.
> > > > > >
> > > > > > It would also not match how the systems that actually produce
> these
> > > > > vectors
> > > > > > treat floats. Most of them use IEEE floats, e.g. Spark, DuckDB,
> > > NumPy,
> > > > > > PyTorch, TensorFlow, or lower level CUDA, oneMKL, and OpenBLAS.
> > > > > >
> > > > > > pgvector's "all elements must be finite" rule is an insert
> policy.
> > > > While
> > > > > I
> > > > > > think Parquet as a general-purpose format should allow them out
> of
> > > the
> > > > > box.
> > > > > > It is a user system decision to disallow writing these values or
> to
> > > > throw
> > > > > > errors when encountering them.
> > > > > >
> > > > > > A type-level ban also does not keep Inf/NaN out of vector math.
> > > Finite
> > > > > > coordinates still overflow. For instance: magnitude =
> ||[FLOAT_MAX,
> > > > > > FLOAT_MAX]|| where every coordinate is finite overflows to +Inf.
> > > > PyTorch
> > > > > > documents the same overflow: tensor([1e20, 1e20]).norm() is inf
> in
> > > > fp32.
> > > > > >
> > > > > > A generated MAGNITUDE(vec) column would then be Inf while the
> > VECTOR
> > > > > itself
> > > > > > was required to be finite. The ban only forbids storing specials
> in
> > > the
> > > > > > coordinates, and does not prevent even simple downstream
> > computations
> > > > > from
> > > > > > producing them.
> > > > > >
> > > > > > I suggest keeping VECTOR as the type and add an optional bool
> > > > > > VectorType.finite (default false). Engines that want
> pgvector-like
> > > > > > rejection can require the flag = true. Engines that use NumPy /
> > > > PyTorch /
> > > > > > Spark arrays can keep representable IEEE values. For the same
> > > reason, I
> > > > > do
> > > > > > not favor adding yet another type called FINITE_VECTOR.
> > > > > >
> > > > > > Best,
> > > > > > Sebastian
> > > > > >
> > > > > >
> > > > > > On Tue, Sep 29, 2026 at 2:03 PM Divjot Arora via dev <
> > > > > > dev@parquet.apache.org>
> > > > > > wrote:
> > > > > >
> > > > > > > I agree with moving forward with the proposal and renaming
> VECTOR
> > > to
> > > > > > > FINITE_VECTOR or something
> > > > > > > similar if others feel that better communicates its intent. I
> see
> > > > value
> > > > > > in
> > > > > > > having a vector type that guarantees
> > > > > > > its values are mathematically-correct vectors for a few
> reasons:
> > > > > > >
> > > > > > > 1. As others have mentioned, this is similar to the STRING
> > logical
> > > > type
> > > > > > > mandating UTF-8.
> > > > > > > 2. It is not possible to rely on statistics for detecting NaN
> or
> > > > +-inf
> > > > > > > values as all statistics are optional.
> > > > > > > To my knowledge, there is no precedent for requiring that
> writers
> > > > emit
> > > > > > > stats for specific logical types.
> > > > > > > 3. Without statistics, NaN/inf values could flow through the
> > engine
> > > > and
> > > > > > > block them for specific functionality.
> > > > > > > This seems to be Lance's behavior. However, I don't think this
> is
> > > as
> > > > > easy
> > > > > > > as it sounds and Lance has run
> > > > > > > into several bugs due to this ([1] [2] for examples). Their
> > Python
> > > > [3]
> > > > > > and
> > > > > > > Rust [4] APIs default to erroring on
> > > > > > > vectors that contain NaN values.
> > > > > > >
> > > > > > > I agree that Parquet is a general-purpose format, but I don't
> > think
> > > > > this
> > > > > > > implies the lack of domain-specific types.
> > > > > > > Instead, this broadens the use cases where Parquet would be
> > > > considered
> > > > > a
> > > > > > > good fit. I also fully agree with
> > > > > > > continuing work on a general fixed-size list type if we believe
> > > there
> > > > > are
> > > > > > > non-vector use cases that would benefit
> > > > > > > from such guarantees at the storage layer.
> > > > > > >
> > > > > > > [1] https://github.com/lance-format/lance/pull/3648
> > > > > > > [2] https://github.com/lance-format/lance/issues/8607 (see
> > point 2
> > > > in
> > > > > > > "root
> > > > > > > cause")
> > > > > > > [3]
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/lancedb/lancedb/blob/0327618a62483feb6dca2336549845e5887cd48f/python/python/lancedb/table.py#L4807-L4911
> > > > > > > [4]
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/lancedb/lancedb/blob/0327618a62483feb6dca2336549845e5887cd48f/rust/lancedb/src/table/add_data.rs#L105-L110
> > > > > > >
> > > > > > > On Mon, Sep 28, 2026 at 6:19 PM Philipp Fischbeck via dev <
> > > > > > > dev@parquet.apache.org> wrote:
> > > > > > >
> > > > > > > > I also favor moving forward with the proposal as is. I see
> > strong
> > > > > value
> > > > > > > and
> > > > > > > > use cases for the numeric vector type, and I agree with the
> > > > > > restrictions
> > > > > > > > around NaN/Inf values and element types in general. Precedent
> > > > exists
> > > > > in
> > > > > > > > other types that restrict which physical type values are
> > > logically
> > > > > > valid
> > > > > > > > (e.g., the geospatial types needing to be WKB-encoded).  A
> > > general
> > > > > > > > fixed-size list also has value, but I think it makes sense to
> > > > > separate
> > > > > > > the
> > > > > > > > two types. The semantics are relevant, and I can see how the
> > two
> > > > > types
> > > > > > > > might diverge further in the future regarding statistics or
> > > > > encodings.
> > > > > > > >
> > > > > > > > I would be okay with renaming the vector type, but it seems
> > like
> > > > the
> > > > > > > VECTOR
> > > > > > > > name is already used across other engines mentioned in this
> > > thread.
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Philipp
> > > > > > > >
> > > > > > > > On Sat, Sep 26, 2026 at 4:37 PM Gang Wu <us...@gmail.com>
> > > wrote:
> > > > > > > >
> > > > > > > > > To separate orthogonal concerns, I feel we should treat the
> > > > general
> > > > > > > > > FixedSizeList as the underlying physical layout, and
> separate
> > > it
> > > > > from
> > > > > > > the
> > > > > > > > > discussion of a specialized VECTOR (or broader ML tensor)
> > > logical
> > > > > > type.
> > > > > > > > >
> > > > > > > > > Currently, the immediate need is to define a specialized
> > VECTOR
> > > > > type
> > > > > > so
> > > > > > > > > that different engines and formats have a common semantic
> > > > contract
> > > > > > for
> > > > > > > > > interoperability.
> > > > > > > > >
> > > > > > > > > If the main point of contention is just the NaN/Inf/null
> > > > > limitations,
> > > > > > > > maybe
> > > > > > > > > we can resolve this by adding a qualification to the name
> > (like
> > > > > > Andrew
> > > > > > > > > suggested, e.g., VECTOR_EMBEDDING or FINITE_VECTOR) to
> avoid
> > > > > > confusion
> > > > > > > > with
> > > > > > > > > broader ML use cases. Alternatively, we could introduce a
> > > boolean
> > > > > > flag
> > > > > > > > > (e.g., optional bool VectorType.finite) as proposed
> earlier.
> > > > > > > > >
> > > > > > > > > On Fri, Sep 25, 2026 at 3:52 AM Andrew Lamb <
> > > > > andrewlamb11@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > I am a little confused about the points of disagreement
> > here.
> > > > > > > > > >
> > > > > > > > > > Is it that the name `VECTOR` is too general for the
> > specific
> > > > use
> > > > > > case
> > > > > > > > of
> > > > > > > > > > fixed length arrays with non-finite values used for
> vector
> > > > > > > embeddings?
> > > > > > > > > If
> > > > > > > > > > so, maybe we could add a qualification to the name
> > > > > (FINITE_VECTOR?
> > > > > > > > > > FEATURE_VECTOR? VECTOR_EMBEDDING?)
> > > > > > > > > >
> > > > > > > > > > It seems like there is precedent for defining logical
> types
> > > > that
> > > > > > > > specify
> > > > > > > > > > certain physical types and restrict their contents to
> help
> > > > > readers
> > > > > > > > > > interpret meaning, for example string[1] or JSON;
> Defining
> > > one
> > > > > for
> > > > > > > > > vector
> > > > > > > > > > embeddings seems reasonable enough to me.
> > > > > > > > > >
> > > > > > > > > > Andrew
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > [1]:
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#string
> > > > > > > > > >
> > > > > > > > > > On Thu, Sep 24, 2026 at 3:34 PM Daniel Weeks <
> > > > dweeks@apache.org>
> > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Well, Parquet is not a single-purpose format.
> > > > > > > > > > > >
> > > > > > > > > > > > Vector stores can choose whatever policy fits their
> > > usage,
> > > > > but
> > > > > > > > we're
> > > > > > > > > > > > talking about a general-purpose file format that aims
> > to
> > > be
> > > > > > > broadly
> > > > > > > > > > > > applicable.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > I agree that the format is intended to be general
> > purpose,
> > > > but
> > > > > > > types
> > > > > > > > > > within
> > > > > > > > > > > the format define specific representation and semantics
> > to
> > > > > > achieve
> > > > > > > > that
> > > > > > > > > > > goal.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > > "Vector" has a more general sense than ML and it's
> > quite
> > > > > > trivial
> > > > > > > to
> > > > > > > > > > > > produce vectors with non-finite values using e.g.
> NumPy
> > > or
> > > > > > Pandas
> > > > > > > > (or
> > > > > > > > > > > > any other piece of dataframe or scientific computing
> > > > > software).
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >  The justification for restricting non-finite values is
> > > > > directly
> > > > > > > > > related
> > > > > > > > > > to
> > > > > > > > > > > the semantic meaning of a vector.  Vectors have
> specific
> > > > > > > > > characteristics
> > > > > > > > > > > (dimension, magnitude, direction, etc).  Non-finite
> > values
> > > > are
> > > > > > > > invalid
> > > > > > > > > > and
> > > > > > > > > > > break those characteristics.  I'd liken it to storing
> > > > non-utf8
> > > > > > > values
> > > > > > > > > in
> > > > > > > > > > a
> > > > > > > > > > > string and still calling it a string.
> > > > > > > > > > >
> > > > > > > > > > > That also differentiates a vector from a fixed length
> > list,
> > > > > which
> > > > > > > is
> > > > > > > > > more
> > > > > > > > > > > general and does not carry the same semantic meaning.
> > > > > > > > > > >
> > > > > > > > > > > I also don't think we need to bundle all similarly
> shaped
> > > use
> > > > > > cases
> > > > > > > > > into
> > > > > > > > > > > the same solution (e.g. flxed size list).  Vector is a
> > well
> > > > > > > > established
> > > > > > > > > > > data type at this point and aligns with the general
> > purpose
> > > > > > nature
> > > > > > > of
> > > > > > > > > the
> > > > > > > > > > > format (many other formats and databases natively
> support
> > > > > this).
> > > > > > > > > > >
> > > > > > > > > > > -Dan
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Thu, Sep 24, 2026 at 11:55 AM Antoine Pitrou <
> > > > > > > antoine@python.org>
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Le 24/09/2026 à 20:11, Russell Spitzer a écrit :
> > > > > > > > > > > > > I think something I haven't seen yet is the clear
> > > > use-case
> > > > > > for
> > > > > > > > > > > non-finite
> > > > > > > > > > > > > values within a vector type. There has been a lot
> of
> > > > > > > theoretical
> > > > > > > > > > > > discussion
> > > > > > > > > > > > > here about use cases but what toolings currently
> > > produce
> > > > > > these
> > > > > > > > > > vectors
> > > > > > > > > > > > and
> > > > > > > > > > > > > how do they function in downstream engines? I'd
> like
> > to
> > > > > limit
> > > > > > > > > > > introducing
> > > > > > > > > > > > > types that have a wide range of values which are
> > simply
> > > > are
> > > > > > > > failure
> > > > > > > > > > > modes
> > > > > > > > > > > > > of different flavors
> > > > > > > > > > > >
> > > > > > > > > > > > "Vector" has a more general sense than ML and it's
> > quite
> > > > > > trivial
> > > > > > > to
> > > > > > > > > > > > produce vectors with non-finite values using e.g.
> NumPy
> > > or
> > > > > > Pandas
> > > > > > > > (or
> > > > > > > > > > > > any other piece of dataframe or scientific computing
> > > > > software).
> > > > > > > > > > > >
> > > > > > > > > > > > It's not like the general concept of a vector is a
> IETF
> > > > > > standard
> > > > > > > or
> > > > > > > > > > > > anything, so analogies with JSON or UTF8 (where the
> > > values
> > > > > are
> > > > > > > > > expected
> > > > > > > > > > > > to be valid according to the respective specs) do not
> > > > apply.
> > > > > > > > > > > >
> > > > > > > > > > > > This discussion would IMHO have a better chance of
> > > > converging
> > > > > > if
> > > > > > > we
> > > > > > > > > > > > separated the two orthogonal concerns of:
> > > > > > > > > > > >
> > > > > > > > > > > > 1. representing fixed-length list/vector data in
> > Parquet
> > > > > (with
> > > > > > a
> > > > > > > > > > > > potential limitation to non-nested types)
> > > > > > > > > > > >
> > > > > > > > > > > > 2. representing the information that a given column
> (or
> > > > > column
> > > > > > > > chunk,
> > > > > > > > > > > > etc.) does not contain non-finite values
> > > > > > > > > > > >
> > > > > > > > > > > > ... both of which are useful on their own right, but
> do
> > > not
> > > > > > have
> > > > > > > to
> > > > > > > > > be
> > > > > > > > > > > > conflated and bundled together.
> > > > > > > > > > > >
> > > > > > > > > > > > Regards
> > > > > > > > > > > >
> > > > > > > > > > > > Antoine.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] Numeric Vector type: physical representation (outcome of focused call)

[Will Edwards via dev <de...@parquet.apache.org>]

Could we clarify the use cases for a VECTOR annotation on LIST<FLOAT>?
Specifically, what does it make possible that's impossible today, and what
does it make easier that's currently hard?

I'm struggling to find a strong answer. I'd like every engine to offer
vector functions on lists of floats without needing a new logical type.

To be clear, I'm still enthusiastic about a vector type that describes how
to unpack and scale composite representations (quantised codes plus scales,
codebooks, etc.), in the direction ML is going. That adds information
engines don't have. A label on LIST<FLOAT> doesn't: engines already
understand the physical type, and many already offer vector functions on
it. I don't see a performance benefit, and NaN/null/Inf checks are cheap
enough that engines should do them anyway. The label also doesn't say what
you can do with the column. Knowing that a column holds embeddings doesn't
say how they can be used, just as zipcode and age are both integers but
multiplying them is meaningless.

My concern is the opposite effect. We're establishing that engines should
ignore logical types they don't understand. So a tool that doesn't know
VECTOR will read the column as LIST<FLOAT>, let the user compute centroids
(say), and write plain LIST<FLOAT> back out. If other engines restrict
embedding functions to VECTOR-annotated columns, that file stops working
with them, through no fault of the user. An annotation that doesn't change
what the data is shouldn't be able to make it unusable.

Am I missing a use case?

Best, Will

On Thu, 8 Oct 2026 at 19:53, Steven Wu <st...@gmail.com> wrote:

> I also think a FINITE_VECTOR type brings a lot of values to many use cases.
> It is would be great to make progress on the FINITE_VECTOR, and not to fit
> every use cases into the FIXED_LENGTH_LIST.
>
> On Mon, Oct 5, 2026 at 1:26 PM Ryan Blue <bl...@apache.org> wrote:
>
> > Going through this thread again, it looks like there are a few things
> that
> > people generally agree on:
> >
> > - In the short term, we want to introduce a logical type or types; a
> better
> > physical representation can be done later. We can focus on logical types.
> > - There is precedent in logical types for restricting values, like UTF8
> > strings to contain only UTF-8 codepoints. I think we also agree that we
> > could restrict vector elements to non-nested or primitive types.
> > - There are many examples of databases that restrict their vector types
> to
> > non-null, finite elements.
> > - Some people want to have write-time guarantees that vectors have only
> > non-null, finite elements
> > - Some people want to extend this work to include tensors, shorter
> floating
> > point representations, variable lengths, NaN or Infinity elements, etc.
> >
> > It sounds like we are nearly there, but have two halves of the community
> > that care about different things. That just means we should have multiple
> > types. It's entirely reasonable to want a type that allows NaN values,
> and
> > it's also reasonable for those of us building databases, table standards,
> > and vector indexes to care about guarantees that elements are finite.
> > There's more than enough examples to show that this is a legitimate use
> > case.
> >
> > Since I don't see the concerns of the different use cases changing, it
> > looks like we need to find a compromise to move forward. I propose that
> we
> > introduce a FINITE_VECTOR type and separately a FIXED_LENGTH_LIST or
> > another VECTOR type that allows infinite and NaN values.
> >
> > Can we agree on a compromise? Is there a strong reason to try to fit
> these
> > use cases into a single type? What if that means not making progress?
> >
> > Ryan
> >
> > On Mon, Oct 5, 2026 at 12:15 PM Philipp Fischbeck via dev <
> > dev@parquet.apache.org> wrote:
> >
> > > Thinking about this more, I am also fine with pulling out the “NaN/Inf”
> > > requirement. Either into a type boolean (my preference), or putting no
> > > restrictions on the Parquet level, allowing all IEEE 754 float numbers.
> > If
> > > some top-level application wants to put additional constraints on the
> > data
> > > type, like Iceberg not allowing NaN/infinity, it can still do so one
> > layer
> > > above. It seems like this was the biggest point of discussion
> remaining.
> > > All these options work for me, and we should move forward with this
> > > proposal.
> > >
> > > Concerning additional element types like bf16, they can be introduced
> as
> > > Parquet types (at least in the current LIST-based spec proposal).
> > >
> > > Concerning Will’s suggestion around representing VECTOR as a variable
> > > length list with a MIN/MAX_LENGTH metadata restriction: I think that a
> > > dedicated logical VECTOR type still provides enough benefits by
> > describing
> > > the semantics of the data, and allowing to capture future changes
> around
> > > statistics. In that case, the length is better captured as an exact
> > length
> > > for this type specifically. I’m also not sure about other use cases for
> > > MIN_LENGTH, so introducing a new type constraint system just for the
> > > MIN_LENGTH=MAX_LENGTH case for vectors doesn’t have enough value in my
> > > view.
> > >
> > > Best,
> > > Philipp
> > >
> > >
> > > On Thu, Oct 1, 2026 at 9:00 AM Will Edwards via dev <
> > > dev@parquet.apache.org>
> > > wrote:
> > >
> > > > +1 Sebastian, a vector type suggests usage beyond normalized
> embeddings
> > > and
> > > > engines really aren't going to be able to elide all the checks
> anyway.
> > > >
> > > > Antoine, you suggested we explore metadata solutions too :)
> > > >
> > > > Today, a database supporting CHAR(N) or VARCHAR(n) can't express this
> > in
> > > > Parquet.  Engines can write opaque blobs they understand in the
> > metadata
> > > > (e.g. Arrow does) so the extra type info round-trips, but this isn't
> > > > standardized, portable, nor interoperable.
> > > >
> > > > So perhaps standardizing constraint metadata or new Thrift fields (my
> > > pref)
> > > > can generically support VECTOR(N) whilst also being more broadly
> useful
> > > for
> > > > other use cases and data types?
> > > >
> > > > Imagine metadata or Thrift fields to tag the MIN_LENGTH and
> MAX_LENGTH
> > > for
> > > > Parquet's variable-length physcial types.  These would count logical
> > > units
> > > > (e.g. UTF8 code points for STRING) rather than bytes.
> > > >
> > > > Then one engine's VECTOR(N) can be written to Parquet as LIST(N) with
> > > > MIN_LENGTH = MAX_LENGTH = N and then readers - even different
> engines -
> > > can
> > > > pick up these constraints if they support doing so.  We don't
> actually
> > > need
> > > > a logical type to express a VECTOR then.
> > > >
> > > > This also allows standard SQL's CHAR(N) and VARCHAR(N) to be
> expressed
> > on
> > > > UTF8 columns etc.
> > > >
> > > > This also seems generally useful in the future as we approach
> > expressing
> > > > other tensor data shapes and types.
> > > >
> > > > This won't change the mechanics: the storage type doesn't enforce
> these
> > > > constraints and engines must check them. It's previously established
> > that
> > > > performance is just a software problem and there is no fundamental
> > > > performance advantage to the fixed size types in Dremel-like formats
> > like
> > > > Parquet.
> > > >
> > > > Best,
> > > > Will
> > > >
> > > > On Wed, 30 Sept 2026 at 16:08, Sebastian Baunsgaard <
> > > baunsgaard@apache.org
> > > > >
> > > > wrote:
> > > >
> > > > > After listening in on the last meeting Wednesday, I have been
> > thinking
> > > > > about this quite a bit, and thought I would share my 2 cents.
> > > > >
> > > > > I suggest not requiring VECTOR elements to be finite, as proposed
> in
> > > the
> > > > PR
> > > > > <https://github.com/apache/parquet-format/pull/624>.
> > > > >
> > > > > While Divjot and others note the specialization for UTF-8, Parquet
> > > > already
> > > > > defines DOUBLE and FLOAT as IEEE encodings
> > > > > <
> > https://github.com/apache/parquet-format/blob/master/README.md#types
> > > >.
> > > > >
> > > > > Making NaN or +/-Inf illegal inside VECTOR would be a new
> restriction
> > > on
> > > > > those same underlying physical types. Scalar FLOAT/DOUBLE columns
> > have
> > > > > never had such a rule.
> > > > >
> > > > > It would also not match how the systems that actually produce these
> > > > vectors
> > > > > treat floats. Most of them use IEEE floats, e.g. Spark, DuckDB,
> > NumPy,
> > > > > PyTorch, TensorFlow, or lower level CUDA, oneMKL, and OpenBLAS.
> > > > >
> > > > > pgvector's "all elements must be finite" rule is an insert policy.
> > > While
> > > > I
> > > > > think Parquet as a general-purpose format should allow them out of
> > the
> > > > box.
> > > > > It is a user system decision to disallow writing these values or to
> > > throw
> > > > > errors when encountering them.
> > > > >
> > > > > A type-level ban also does not keep Inf/NaN out of vector math.
> > Finite
> > > > > coordinates still overflow. For instance: magnitude = ||[FLOAT_MAX,
> > > > > FLOAT_MAX]|| where every coordinate is finite overflows to +Inf.
> > > PyTorch
> > > > > documents the same overflow: tensor([1e20, 1e20]).norm() is inf in
> > > fp32.
> > > > >
> > > > > A generated MAGNITUDE(vec) column would then be Inf while the
> VECTOR
> > > > itself
> > > > > was required to be finite. The ban only forbids storing specials in
> > the
> > > > > coordinates, and does not prevent even simple downstream
> computations
> > > > from
> > > > > producing them.
> > > > >
> > > > > I suggest keeping VECTOR as the type and add an optional bool
> > > > > VectorType.finite (default false). Engines that want pgvector-like
> > > > > rejection can require the flag = true. Engines that use NumPy /
> > > PyTorch /
> > > > > Spark arrays can keep representable IEEE values. For the same
> > reason, I
> > > > do
> > > > > not favor adding yet another type called FINITE_VECTOR.
> > > > >
> > > > > Best,
> > > > > Sebastian
> > > > >
> > > > >
> > > > > On Tue, Sep 29, 2026 at 2:03 PM Divjot Arora via dev <
> > > > > dev@parquet.apache.org>
> > > > > wrote:
> > > > >
> > > > > > I agree with moving forward with the proposal and renaming VECTOR
> > to
> > > > > > FINITE_VECTOR or something
> > > > > > similar if others feel that better communicates its intent. I see
> > > value
> > > > > in
> > > > > > having a vector type that guarantees
> > > > > > its values are mathematically-correct vectors for a few reasons:
> > > > > >
> > > > > > 1. As others have mentioned, this is similar to the STRING
> logical
> > > type
> > > > > > mandating UTF-8.
> > > > > > 2. It is not possible to rely on statistics for detecting NaN or
> > > +-inf
> > > > > > values as all statistics are optional.
> > > > > > To my knowledge, there is no precedent for requiring that writers
> > > emit
> > > > > > stats for specific logical types.
> > > > > > 3. Without statistics, NaN/inf values could flow through the
> engine
> > > and
> > > > > > block them for specific functionality.
> > > > > > This seems to be Lance's behavior. However, I don't think this is
> > as
> > > > easy
> > > > > > as it sounds and Lance has run
> > > > > > into several bugs due to this ([1] [2] for examples). Their
> Python
> > > [3]
> > > > > and
> > > > > > Rust [4] APIs default to erroring on
> > > > > > vectors that contain NaN values.
> > > > > >
> > > > > > I agree that Parquet is a general-purpose format, but I don't
> think
> > > > this
> > > > > > implies the lack of domain-specific types.
> > > > > > Instead, this broadens the use cases where Parquet would be
> > > considered
> > > > a
> > > > > > good fit. I also fully agree with
> > > > > > continuing work on a general fixed-size list type if we believe
> > there
> > > > are
> > > > > > non-vector use cases that would benefit
> > > > > > from such guarantees at the storage layer.
> > > > > >
> > > > > > [1] https://github.com/lance-format/lance/pull/3648
> > > > > > [2] https://github.com/lance-format/lance/issues/8607 (see
> point 2
> > > in
> > > > > > "root
> > > > > > cause")
> > > > > > [3]
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/lancedb/lancedb/blob/0327618a62483feb6dca2336549845e5887cd48f/python/python/lancedb/table.py#L4807-L4911
> > > > > > [4]
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/lancedb/lancedb/blob/0327618a62483feb6dca2336549845e5887cd48f/rust/lancedb/src/table/add_data.rs#L105-L110
> > > > > >
> > > > > > On Mon, Sep 28, 2026 at 6:19 PM Philipp Fischbeck via dev <
> > > > > > dev@parquet.apache.org> wrote:
> > > > > >
> > > > > > > I also favor moving forward with the proposal as is. I see
> strong
> > > > value
> > > > > > and
> > > > > > > use cases for the numeric vector type, and I agree with the
> > > > > restrictions
> > > > > > > around NaN/Inf values and element types in general. Precedent
> > > exists
> > > > in
> > > > > > > other types that restrict which physical type values are
> > logically
> > > > > valid
> > > > > > > (e.g., the geospatial types needing to be WKB-encoded).  A
> > general
> > > > > > > fixed-size list also has value, but I think it makes sense to
> > > > separate
> > > > > > the
> > > > > > > two types. The semantics are relevant, and I can see how the
> two
> > > > types
> > > > > > > might diverge further in the future regarding statistics or
> > > > encodings.
> > > > > > >
> > > > > > > I would be okay with renaming the vector type, but it seems
> like
> > > the
> > > > > > VECTOR
> > > > > > > name is already used across other engines mentioned in this
> > thread.
> > > > > > >
> > > > > > > Best,
> > > > > > > Philipp
> > > > > > >
> > > > > > > On Sat, Sep 26, 2026 at 4:37 PM Gang Wu <us...@gmail.com>
> > wrote:
> > > > > > >
> > > > > > > > To separate orthogonal concerns, I feel we should treat the
> > > general
> > > > > > > > FixedSizeList as the underlying physical layout, and separate
> > it
> > > > from
> > > > > > the
> > > > > > > > discussion of a specialized VECTOR (or broader ML tensor)
> > logical
> > > > > type.
> > > > > > > >
> > > > > > > > Currently, the immediate need is to define a specialized
> VECTOR
> > > > type
> > > > > so
> > > > > > > > that different engines and formats have a common semantic
> > > contract
> > > > > for
> > > > > > > > interoperability.
> > > > > > > >
> > > > > > > > If the main point of contention is just the NaN/Inf/null
> > > > limitations,
> > > > > > > maybe
> > > > > > > > we can resolve this by adding a qualification to the name
> (like
> > > > > Andrew
> > > > > > > > suggested, e.g., VECTOR_EMBEDDING or FINITE_VECTOR) to avoid
> > > > > confusion
> > > > > > > with
> > > > > > > > broader ML use cases. Alternatively, we could introduce a
> > boolean
> > > > > flag
> > > > > > > > (e.g., optional bool VectorType.finite) as proposed earlier.
> > > > > > > >
> > > > > > > > On Fri, Sep 25, 2026 at 3:52 AM Andrew Lamb <
> > > > andrewlamb11@gmail.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > I am a little confused about the points of disagreement
> here.
> > > > > > > > >
> > > > > > > > > Is it that the name `VECTOR` is too general for the
> specific
> > > use
> > > > > case
> > > > > > > of
> > > > > > > > > fixed length arrays with non-finite values used for vector
> > > > > > embeddings?
> > > > > > > > If
> > > > > > > > > so, maybe we could add a qualification to the name
> > > > (FINITE_VECTOR?
> > > > > > > > > FEATURE_VECTOR? VECTOR_EMBEDDING?)
> > > > > > > > >
> > > > > > > > > It seems like there is precedent for defining logical types
> > > that
> > > > > > > specify
> > > > > > > > > certain physical types and restrict their contents to help
> > > > readers
> > > > > > > > > interpret meaning, for example string[1] or JSON;  Defining
> > one
> > > > for
> > > > > > > > vector
> > > > > > > > > embeddings seems reasonable enough to me.
> > > > > > > > >
> > > > > > > > > Andrew
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > [1]:
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#string
> > > > > > > > >
> > > > > > > > > On Thu, Sep 24, 2026 at 3:34 PM Daniel Weeks <
> > > dweeks@apache.org>
> > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Well, Parquet is not a single-purpose format.
> > > > > > > > > > >
> > > > > > > > > > > Vector stores can choose whatever policy fits their
> > usage,
> > > > but
> > > > > > > we're
> > > > > > > > > > > talking about a general-purpose file format that aims
> to
> > be
> > > > > > broadly
> > > > > > > > > > > applicable.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > I agree that the format is intended to be general
> purpose,
> > > but
> > > > > > types
> > > > > > > > > within
> > > > > > > > > > the format define specific representation and semantics
> to
> > > > > achieve
> > > > > > > that
> > > > > > > > > > goal.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > > "Vector" has a more general sense than ML and it's
> quite
> > > > > trivial
> > > > > > to
> > > > > > > > > > > produce vectors with non-finite values using e.g. NumPy
> > or
> > > > > Pandas
> > > > > > > (or
> > > > > > > > > > > any other piece of dataframe or scientific computing
> > > > software).
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >  The justification for restricting non-finite values is
> > > > directly
> > > > > > > > related
> > > > > > > > > to
> > > > > > > > > > the semantic meaning of a vector.  Vectors have specific
> > > > > > > > characteristics
> > > > > > > > > > (dimension, magnitude, direction, etc).  Non-finite
> values
> > > are
> > > > > > > invalid
> > > > > > > > > and
> > > > > > > > > > break those characteristics.  I'd liken it to storing
> > > non-utf8
> > > > > > values
> > > > > > > > in
> > > > > > > > > a
> > > > > > > > > > string and still calling it a string.
> > > > > > > > > >
> > > > > > > > > > That also differentiates a vector from a fixed length
> list,
> > > > which
> > > > > > is
> > > > > > > > more
> > > > > > > > > > general and does not carry the same semantic meaning.
> > > > > > > > > >
> > > > > > > > > > I also don't think we need to bundle all similarly shaped
> > use
> > > > > cases
> > > > > > > > into
> > > > > > > > > > the same solution (e.g. flxed size list).  Vector is a
> well
> > > > > > > established
> > > > > > > > > > data type at this point and aligns with the general
> purpose
> > > > > nature
> > > > > > of
> > > > > > > > the
> > > > > > > > > > format (many other formats and databases natively support
> > > > this).
> > > > > > > > > >
> > > > > > > > > > -Dan
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Thu, Sep 24, 2026 at 11:55 AM Antoine Pitrou <
> > > > > > antoine@python.org>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Le 24/09/2026 à 20:11, Russell Spitzer a écrit :
> > > > > > > > > > > > I think something I haven't seen yet is the clear
> > > use-case
> > > > > for
> > > > > > > > > > non-finite
> > > > > > > > > > > > values within a vector type. There has been a lot of
> > > > > > theoretical
> > > > > > > > > > > discussion
> > > > > > > > > > > > here about use cases but what toolings currently
> > produce
> > > > > these
> > > > > > > > > vectors
> > > > > > > > > > > and
> > > > > > > > > > > > how do they function in downstream engines? I'd like
> to
> > > > limit
> > > > > > > > > > introducing
> > > > > > > > > > > > types that have a wide range of values which are
> simply
> > > are
> > > > > > > failure
> > > > > > > > > > modes
> > > > > > > > > > > > of different flavors
> > > > > > > > > > >
> > > > > > > > > > > "Vector" has a more general sense than ML and it's
> quite
> > > > > trivial
> > > > > > to
> > > > > > > > > > > produce vectors with non-finite values using e.g. NumPy
> > or
> > > > > Pandas
> > > > > > > (or
> > > > > > > > > > > any other piece of dataframe or scientific computing
> > > > software).
> > > > > > > > > > >
> > > > > > > > > > > It's not like the general concept of a vector is a IETF
> > > > > standard
> > > > > > or
> > > > > > > > > > > anything, so analogies with JSON or UTF8 (where the
> > values
> > > > are
> > > > > > > > expected
> > > > > > > > > > > to be valid according to the respective specs) do not
> > > apply.
> > > > > > > > > > >
> > > > > > > > > > > This discussion would IMHO have a better chance of
> > > converging
> > > > > if
> > > > > > we
> > > > > > > > > > > separated the two orthogonal concerns of:
> > > > > > > > > > >
> > > > > > > > > > > 1. representing fixed-length list/vector data in
> Parquet
> > > > (with
> > > > > a
> > > > > > > > > > > potential limitation to non-nested types)
> > > > > > > > > > >
> > > > > > > > > > > 2. representing the information that a given column (or
> > > > column
> > > > > > > chunk,
> > > > > > > > > > > etc.) does not contain non-finite values
> > > > > > > > > > >
> > > > > > > > > > > ... both of which are useful on their own right, but do
> > not
> > > > > have
> > > > > > to
> > > > > > > > be
> > > > > > > > > > > conflated and bundled together.
> > > > > > > > > > >
> > > > > > > > > > > Regards
> > > > > > > > > > >
> > > > > > > > > > > Antoine.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] Numeric Vector type: physical representation (outcome of focused call)

[Steven Wu <st...@gmail.com>]

I also think a FINITE_VECTOR type brings a lot of values to many use cases.
It is would be great to make progress on the FINITE_VECTOR, and not to fit
every use cases into the FIXED_LENGTH_LIST.

On Mon, Oct 5, 2026 at 1:26 PM Ryan Blue <bl...@apache.org> wrote:

> Going through this thread again, it looks like there are a few things that
> people generally agree on:
>
> - In the short term, we want to introduce a logical type or types; a better
> physical representation can be done later. We can focus on logical types.
> - There is precedent in logical types for restricting values, like UTF8
> strings to contain only UTF-8 codepoints. I think we also agree that we
> could restrict vector elements to non-nested or primitive types.
> - There are many examples of databases that restrict their vector types to
> non-null, finite elements.
> - Some people want to have write-time guarantees that vectors have only
> non-null, finite elements
> - Some people want to extend this work to include tensors, shorter floating
> point representations, variable lengths, NaN or Infinity elements, etc.
>
> It sounds like we are nearly there, but have two halves of the community
> that care about different things. That just means we should have multiple
> types. It's entirely reasonable to want a type that allows NaN values, and
> it's also reasonable for those of us building databases, table standards,
> and vector indexes to care about guarantees that elements are finite.
> There's more than enough examples to show that this is a legitimate use
> case.
>
> Since I don't see the concerns of the different use cases changing, it
> looks like we need to find a compromise to move forward. I propose that we
> introduce a FINITE_VECTOR type and separately a FIXED_LENGTH_LIST or
> another VECTOR type that allows infinite and NaN values.
>
> Can we agree on a compromise? Is there a strong reason to try to fit these
> use cases into a single type? What if that means not making progress?
>
> Ryan
>
> On Mon, Oct 5, 2026 at 12:15 PM Philipp Fischbeck via dev <
> dev@parquet.apache.org> wrote:
>
> > Thinking about this more, I am also fine with pulling out the “NaN/Inf”
> > requirement. Either into a type boolean (my preference), or putting no
> > restrictions on the Parquet level, allowing all IEEE 754 float numbers.
> If
> > some top-level application wants to put additional constraints on the
> data
> > type, like Iceberg not allowing NaN/infinity, it can still do so one
> layer
> > above. It seems like this was the biggest point of discussion remaining.
> > All these options work for me, and we should move forward with this
> > proposal.
> >
> > Concerning additional element types like bf16, they can be introduced as
> > Parquet types (at least in the current LIST-based spec proposal).
> >
> > Concerning Will’s suggestion around representing VECTOR as a variable
> > length list with a MIN/MAX_LENGTH metadata restriction: I think that a
> > dedicated logical VECTOR type still provides enough benefits by
> describing
> > the semantics of the data, and allowing to capture future changes around
> > statistics. In that case, the length is better captured as an exact
> length
> > for this type specifically. I’m also not sure about other use cases for
> > MIN_LENGTH, so introducing a new type constraint system just for the
> > MIN_LENGTH=MAX_LENGTH case for vectors doesn’t have enough value in my
> > view.
> >
> > Best,
> > Philipp
> >
> >
> > On Thu, Oct 1, 2026 at 9:00 AM Will Edwards via dev <
> > dev@parquet.apache.org>
> > wrote:
> >
> > > +1 Sebastian, a vector type suggests usage beyond normalized embeddings
> > and
> > > engines really aren't going to be able to elide all the checks anyway.
> > >
> > > Antoine, you suggested we explore metadata solutions too :)
> > >
> > > Today, a database supporting CHAR(N) or VARCHAR(n) can't express this
> in
> > > Parquet.  Engines can write opaque blobs they understand in the
> metadata
> > > (e.g. Arrow does) so the extra type info round-trips, but this isn't
> > > standardized, portable, nor interoperable.
> > >
> > > So perhaps standardizing constraint metadata or new Thrift fields (my
> > pref)
> > > can generically support VECTOR(N) whilst also being more broadly useful
> > for
> > > other use cases and data types?
> > >
> > > Imagine metadata or Thrift fields to tag the MIN_LENGTH and MAX_LENGTH
> > for
> > > Parquet's variable-length physcial types.  These would count logical
> > units
> > > (e.g. UTF8 code points for STRING) rather than bytes.
> > >
> > > Then one engine's VECTOR(N) can be written to Parquet as LIST(N) with
> > > MIN_LENGTH = MAX_LENGTH = N and then readers - even different engines -
> > can
> > > pick up these constraints if they support doing so.  We don't actually
> > need
> > > a logical type to express a VECTOR then.
> > >
> > > This also allows standard SQL's CHAR(N) and VARCHAR(N) to be expressed
> on
> > > UTF8 columns etc.
> > >
> > > This also seems generally useful in the future as we approach
> expressing
> > > other tensor data shapes and types.
> > >
> > > This won't change the mechanics: the storage type doesn't enforce these
> > > constraints and engines must check them. It's previously established
> that
> > > performance is just a software problem and there is no fundamental
> > > performance advantage to the fixed size types in Dremel-like formats
> like
> > > Parquet.
> > >
> > > Best,
> > > Will
> > >
> > > On Wed, 30 Sept 2026 at 16:08, Sebastian Baunsgaard <
> > baunsgaard@apache.org
> > > >
> > > wrote:
> > >
> > > > After listening in on the last meeting Wednesday, I have been
> thinking
> > > > about this quite a bit, and thought I would share my 2 cents.
> > > >
> > > > I suggest not requiring VECTOR elements to be finite, as proposed in
> > the
> > > PR
> > > > <https://github.com/apache/parquet-format/pull/624>.
> > > >
> > > > While Divjot and others note the specialization for UTF-8, Parquet
> > > already
> > > > defines DOUBLE and FLOAT as IEEE encodings
> > > > <
> https://github.com/apache/parquet-format/blob/master/README.md#types
> > >.
> > > >
> > > > Making NaN or +/-Inf illegal inside VECTOR would be a new restriction
> > on
> > > > those same underlying physical types. Scalar FLOAT/DOUBLE columns
> have
> > > > never had such a rule.
> > > >
> > > > It would also not match how the systems that actually produce these
> > > vectors
> > > > treat floats. Most of them use IEEE floats, e.g. Spark, DuckDB,
> NumPy,
> > > > PyTorch, TensorFlow, or lower level CUDA, oneMKL, and OpenBLAS.
> > > >
> > > > pgvector's "all elements must be finite" rule is an insert policy.
> > While
> > > I
> > > > think Parquet as a general-purpose format should allow them out of
> the
> > > box.
> > > > It is a user system decision to disallow writing these values or to
> > throw
> > > > errors when encountering them.
> > > >
> > > > A type-level ban also does not keep Inf/NaN out of vector math.
> Finite
> > > > coordinates still overflow. For instance: magnitude = ||[FLOAT_MAX,
> > > > FLOAT_MAX]|| where every coordinate is finite overflows to +Inf.
> > PyTorch
> > > > documents the same overflow: tensor([1e20, 1e20]).norm() is inf in
> > fp32.
> > > >
> > > > A generated MAGNITUDE(vec) column would then be Inf while the VECTOR
> > > itself
> > > > was required to be finite. The ban only forbids storing specials in
> the
> > > > coordinates, and does not prevent even simple downstream computations
> > > from
> > > > producing them.
> > > >
> > > > I suggest keeping VECTOR as the type and add an optional bool
> > > > VectorType.finite (default false). Engines that want pgvector-like
> > > > rejection can require the flag = true. Engines that use NumPy /
> > PyTorch /
> > > > Spark arrays can keep representable IEEE values. For the same
> reason, I
> > > do
> > > > not favor adding yet another type called FINITE_VECTOR.
> > > >
> > > > Best,
> > > > Sebastian
> > > >
> > > >
> > > > On Tue, Sep 29, 2026 at 2:03 PM Divjot Arora via dev <
> > > > dev@parquet.apache.org>
> > > > wrote:
> > > >
> > > > > I agree with moving forward with the proposal and renaming VECTOR
> to
> > > > > FINITE_VECTOR or something
> > > > > similar if others feel that better communicates its intent. I see
> > value
> > > > in
> > > > > having a vector type that guarantees
> > > > > its values are mathematically-correct vectors for a few reasons:
> > > > >
> > > > > 1. As others have mentioned, this is similar to the STRING logical
> > type
> > > > > mandating UTF-8.
> > > > > 2. It is not possible to rely on statistics for detecting NaN or
> > +-inf
> > > > > values as all statistics are optional.
> > > > > To my knowledge, there is no precedent for requiring that writers
> > emit
> > > > > stats for specific logical types.
> > > > > 3. Without statistics, NaN/inf values could flow through the engine
> > and
> > > > > block them for specific functionality.
> > > > > This seems to be Lance's behavior. However, I don't think this is
> as
> > > easy
> > > > > as it sounds and Lance has run
> > > > > into several bugs due to this ([1] [2] for examples). Their Python
> > [3]
> > > > and
> > > > > Rust [4] APIs default to erroring on
> > > > > vectors that contain NaN values.
> > > > >
> > > > > I agree that Parquet is a general-purpose format, but I don't think
> > > this
> > > > > implies the lack of domain-specific types.
> > > > > Instead, this broadens the use cases where Parquet would be
> > considered
> > > a
> > > > > good fit. I also fully agree with
> > > > > continuing work on a general fixed-size list type if we believe
> there
> > > are
> > > > > non-vector use cases that would benefit
> > > > > from such guarantees at the storage layer.
> > > > >
> > > > > [1] https://github.com/lance-format/lance/pull/3648
> > > > > [2] https://github.com/lance-format/lance/issues/8607 (see point 2
> > in
> > > > > "root
> > > > > cause")
> > > > > [3]
> > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/lancedb/lancedb/blob/0327618a62483feb6dca2336549845e5887cd48f/python/python/lancedb/table.py#L4807-L4911
> > > > > [4]
> > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/lancedb/lancedb/blob/0327618a62483feb6dca2336549845e5887cd48f/rust/lancedb/src/table/add_data.rs#L105-L110
> > > > >
> > > > > On Mon, Sep 28, 2026 at 6:19 PM Philipp Fischbeck via dev <
> > > > > dev@parquet.apache.org> wrote:
> > > > >
> > > > > > I also favor moving forward with the proposal as is. I see strong
> > > value
> > > > > and
> > > > > > use cases for the numeric vector type, and I agree with the
> > > > restrictions
> > > > > > around NaN/Inf values and element types in general. Precedent
> > exists
> > > in
> > > > > > other types that restrict which physical type values are
> logically
> > > > valid
> > > > > > (e.g., the geospatial types needing to be WKB-encoded).  A
> general
> > > > > > fixed-size list also has value, but I think it makes sense to
> > > separate
> > > > > the
> > > > > > two types. The semantics are relevant, and I can see how the two
> > > types
> > > > > > might diverge further in the future regarding statistics or
> > > encodings.
> > > > > >
> > > > > > I would be okay with renaming the vector type, but it seems like
> > the
> > > > > VECTOR
> > > > > > name is already used across other engines mentioned in this
> thread.
> > > > > >
> > > > > > Best,
> > > > > > Philipp
> > > > > >
> > > > > > On Sat, Sep 26, 2026 at 4:37 PM Gang Wu <us...@gmail.com>
> wrote:
> > > > > >
> > > > > > > To separate orthogonal concerns, I feel we should treat the
> > general
> > > > > > > FixedSizeList as the underlying physical layout, and separate
> it
> > > from
> > > > > the
> > > > > > > discussion of a specialized VECTOR (or broader ML tensor)
> logical
> > > > type.
> > > > > > >
> > > > > > > Currently, the immediate need is to define a specialized VECTOR
> > > type
> > > > so
> > > > > > > that different engines and formats have a common semantic
> > contract
> > > > for
> > > > > > > interoperability.
> > > > > > >
> > > > > > > If the main point of contention is just the NaN/Inf/null
> > > limitations,
> > > > > > maybe
> > > > > > > we can resolve this by adding a qualification to the name (like
> > > > Andrew
> > > > > > > suggested, e.g., VECTOR_EMBEDDING or FINITE_VECTOR) to avoid
> > > > confusion
> > > > > > with
> > > > > > > broader ML use cases. Alternatively, we could introduce a
> boolean
> > > > flag
> > > > > > > (e.g., optional bool VectorType.finite) as proposed earlier.
> > > > > > >
> > > > > > > On Fri, Sep 25, 2026 at 3:52 AM Andrew Lamb <
> > > andrewlamb11@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > I am a little confused about the points of disagreement here.
> > > > > > > >
> > > > > > > > Is it that the name `VECTOR` is too general for the specific
> > use
> > > > case
> > > > > > of
> > > > > > > > fixed length arrays with non-finite values used for vector
> > > > > embeddings?
> > > > > > > If
> > > > > > > > so, maybe we could add a qualification to the name
> > > (FINITE_VECTOR?
> > > > > > > > FEATURE_VECTOR? VECTOR_EMBEDDING?)
> > > > > > > >
> > > > > > > > It seems like there is precedent for defining logical types
> > that
> > > > > > specify
> > > > > > > > certain physical types and restrict their contents to help
> > > readers
> > > > > > > > interpret meaning, for example string[1] or JSON;  Defining
> one
> > > for
> > > > > > > vector
> > > > > > > > embeddings seems reasonable enough to me.
> > > > > > > >
> > > > > > > > Andrew
> > > > > > > >
> > > > > > > >
> > > > > > > > [1]:
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#string
> > > > > > > >
> > > > > > > > On Thu, Sep 24, 2026 at 3:34 PM Daniel Weeks <
> > dweeks@apache.org>
> > > > > > wrote:
> > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Well, Parquet is not a single-purpose format.
> > > > > > > > > >
> > > > > > > > > > Vector stores can choose whatever policy fits their
> usage,
> > > but
> > > > > > we're
> > > > > > > > > > talking about a general-purpose file format that aims to
> be
> > > > > broadly
> > > > > > > > > > applicable.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > I agree that the format is intended to be general purpose,
> > but
> > > > > types
> > > > > > > > within
> > > > > > > > > the format define specific representation and semantics to
> > > > achieve
> > > > > > that
> > > > > > > > > goal.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > > "Vector" has a more general sense than ML and it's quite
> > > > trivial
> > > > > to
> > > > > > > > > > produce vectors with non-finite values using e.g. NumPy
> or
> > > > Pandas
> > > > > > (or
> > > > > > > > > > any other piece of dataframe or scientific computing
> > > software).
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >  The justification for restricting non-finite values is
> > > directly
> > > > > > > related
> > > > > > > > to
> > > > > > > > > the semantic meaning of a vector.  Vectors have specific
> > > > > > > characteristics
> > > > > > > > > (dimension, magnitude, direction, etc).  Non-finite values
> > are
> > > > > > invalid
> > > > > > > > and
> > > > > > > > > break those characteristics.  I'd liken it to storing
> > non-utf8
> > > > > values
> > > > > > > in
> > > > > > > > a
> > > > > > > > > string and still calling it a string.
> > > > > > > > >
> > > > > > > > > That also differentiates a vector from a fixed length list,
> > > which
> > > > > is
> > > > > > > more
> > > > > > > > > general and does not carry the same semantic meaning.
> > > > > > > > >
> > > > > > > > > I also don't think we need to bundle all similarly shaped
> use
> > > > cases
> > > > > > > into
> > > > > > > > > the same solution (e.g. flxed size list).  Vector is a well
> > > > > > established
> > > > > > > > > data type at this point and aligns with the general purpose
> > > > nature
> > > > > of
> > > > > > > the
> > > > > > > > > format (many other formats and databases natively support
> > > this).
> > > > > > > > >
> > > > > > > > > -Dan
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Thu, Sep 24, 2026 at 11:55 AM Antoine Pitrou <
> > > > > antoine@python.org>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Le 24/09/2026 à 20:11, Russell Spitzer a écrit :
> > > > > > > > > > > I think something I haven't seen yet is the clear
> > use-case
> > > > for
> > > > > > > > > non-finite
> > > > > > > > > > > values within a vector type. There has been a lot of
> > > > > theoretical
> > > > > > > > > > discussion
> > > > > > > > > > > here about use cases but what toolings currently
> produce
> > > > these
> > > > > > > > vectors
> > > > > > > > > > and
> > > > > > > > > > > how do they function in downstream engines? I'd like to
> > > limit
> > > > > > > > > introducing
> > > > > > > > > > > types that have a wide range of values which are simply
> > are
> > > > > > failure
> > > > > > > > > modes
> > > > > > > > > > > of different flavors
> > > > > > > > > >
> > > > > > > > > > "Vector" has a more general sense than ML and it's quite
> > > > trivial
> > > > > to
> > > > > > > > > > produce vectors with non-finite values using e.g. NumPy
> or
> > > > Pandas
> > > > > > (or
> > > > > > > > > > any other piece of dataframe or scientific computing
> > > software).
> > > > > > > > > >
> > > > > > > > > > It's not like the general concept of a vector is a IETF
> > > > standard
> > > > > or
> > > > > > > > > > anything, so analogies with JSON or UTF8 (where the
> values
> > > are
> > > > > > > expected
> > > > > > > > > > to be valid according to the respective specs) do not
> > apply.
> > > > > > > > > >
> > > > > > > > > > This discussion would IMHO have a better chance of
> > converging
> > > > if
> > > > > we
> > > > > > > > > > separated the two orthogonal concerns of:
> > > > > > > > > >
> > > > > > > > > > 1. representing fixed-length list/vector data in Parquet
> > > (with
> > > > a
> > > > > > > > > > potential limitation to non-nested types)
> > > > > > > > > >
> > > > > > > > > > 2. representing the information that a given column (or
> > > column
> > > > > > chunk,
> > > > > > > > > > etc.) does not contain non-finite values
> > > > > > > > > >
> > > > > > > > > > ... both of which are useful on their own right, but do
> not
> > > > have
> > > > > to
> > > > > > > be
> > > > > > > > > > conflated and bundled together.
> > > > > > > > > >
> > > > > > > > > > Regards
> > > > > > > > > >
> > > > > > > > > > Antoine.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] Numeric Vector type: physical representation (outcome of focused call)

[Ryan Blue <bl...@apache.org>]

Going through this thread again, it looks like there are a few things that
people generally agree on:

- In the short term, we want to introduce a logical type or types; a better
physical representation can be done later. We can focus on logical types.
- There is precedent in logical types for restricting values, like UTF8
strings to contain only UTF-8 codepoints. I think we also agree that we
could restrict vector elements to non-nested or primitive types.
- There are many examples of databases that restrict their vector types to
non-null, finite elements.
- Some people want to have write-time guarantees that vectors have only
non-null, finite elements
- Some people want to extend this work to include tensors, shorter floating
point representations, variable lengths, NaN or Infinity elements, etc.

It sounds like we are nearly there, but have two halves of the community
that care about different things. That just means we should have multiple
types. It's entirely reasonable to want a type that allows NaN values, and
it's also reasonable for those of us building databases, table standards,
and vector indexes to care about guarantees that elements are finite.
There's more than enough examples to show that this is a legitimate use
case.

Since I don't see the concerns of the different use cases changing, it
looks like we need to find a compromise to move forward. I propose that we
introduce a FINITE_VECTOR type and separately a FIXED_LENGTH_LIST or
another VECTOR type that allows infinite and NaN values.

Can we agree on a compromise? Is there a strong reason to try to fit these
use cases into a single type? What if that means not making progress?

Ryan

On Mon, Oct 5, 2026 at 12:15 PM Philipp Fischbeck via dev <
dev@parquet.apache.org> wrote:

> Thinking about this more, I am also fine with pulling out the “NaN/Inf”
> requirement. Either into a type boolean (my preference), or putting no
> restrictions on the Parquet level, allowing all IEEE 754 float numbers. If
> some top-level application wants to put additional constraints on the data
> type, like Iceberg not allowing NaN/infinity, it can still do so one layer
> above. It seems like this was the biggest point of discussion remaining.
> All these options work for me, and we should move forward with this
> proposal.
>
> Concerning additional element types like bf16, they can be introduced as
> Parquet types (at least in the current LIST-based spec proposal).
>
> Concerning Will’s suggestion around representing VECTOR as a variable
> length list with a MIN/MAX_LENGTH metadata restriction: I think that a
> dedicated logical VECTOR type still provides enough benefits by describing
> the semantics of the data, and allowing to capture future changes around
> statistics. In that case, the length is better captured as an exact length
> for this type specifically. I’m also not sure about other use cases for
> MIN_LENGTH, so introducing a new type constraint system just for the
> MIN_LENGTH=MAX_LENGTH case for vectors doesn’t have enough value in my
> view.
>
> Best,
> Philipp
>
>
> On Thu, Oct 1, 2026 at 9:00 AM Will Edwards via dev <
> dev@parquet.apache.org>
> wrote:
>
> > +1 Sebastian, a vector type suggests usage beyond normalized embeddings
> and
> > engines really aren't going to be able to elide all the checks anyway.
> >
> > Antoine, you suggested we explore metadata solutions too :)
> >
> > Today, a database supporting CHAR(N) or VARCHAR(n) can't express this in
> > Parquet.  Engines can write opaque blobs they understand in the metadata
> > (e.g. Arrow does) so the extra type info round-trips, but this isn't
> > standardized, portable, nor interoperable.
> >
> > So perhaps standardizing constraint metadata or new Thrift fields (my
> pref)
> > can generically support VECTOR(N) whilst also being more broadly useful
> for
> > other use cases and data types?
> >
> > Imagine metadata or Thrift fields to tag the MIN_LENGTH and MAX_LENGTH
> for
> > Parquet's variable-length physcial types.  These would count logical
> units
> > (e.g. UTF8 code points for STRING) rather than bytes.
> >
> > Then one engine's VECTOR(N) can be written to Parquet as LIST(N) with
> > MIN_LENGTH = MAX_LENGTH = N and then readers - even different engines -
> can
> > pick up these constraints if they support doing so.  We don't actually
> need
> > a logical type to express a VECTOR then.
> >
> > This also allows standard SQL's CHAR(N) and VARCHAR(N) to be expressed on
> > UTF8 columns etc.
> >
> > This also seems generally useful in the future as we approach expressing
> > other tensor data shapes and types.
> >
> > This won't change the mechanics: the storage type doesn't enforce these
> > constraints and engines must check them. It's previously established that
> > performance is just a software problem and there is no fundamental
> > performance advantage to the fixed size types in Dremel-like formats like
> > Parquet.
> >
> > Best,
> > Will
> >
> > On Wed, 30 Sept 2026 at 16:08, Sebastian Baunsgaard <
> baunsgaard@apache.org
> > >
> > wrote:
> >
> > > After listening in on the last meeting Wednesday, I have been thinking
> > > about this quite a bit, and thought I would share my 2 cents.
> > >
> > > I suggest not requiring VECTOR elements to be finite, as proposed in
> the
> > PR
> > > <https://github.com/apache/parquet-format/pull/624>.
> > >
> > > While Divjot and others note the specialization for UTF-8, Parquet
> > already
> > > defines DOUBLE and FLOAT as IEEE encodings
> > > <https://github.com/apache/parquet-format/blob/master/README.md#types
> >.
> > >
> > > Making NaN or +/-Inf illegal inside VECTOR would be a new restriction
> on
> > > those same underlying physical types. Scalar FLOAT/DOUBLE columns have
> > > never had such a rule.
> > >
> > > It would also not match how the systems that actually produce these
> > vectors
> > > treat floats. Most of them use IEEE floats, e.g. Spark, DuckDB, NumPy,
> > > PyTorch, TensorFlow, or lower level CUDA, oneMKL, and OpenBLAS.
> > >
> > > pgvector's "all elements must be finite" rule is an insert policy.
> While
> > I
> > > think Parquet as a general-purpose format should allow them out of the
> > box.
> > > It is a user system decision to disallow writing these values or to
> throw
> > > errors when encountering them.
> > >
> > > A type-level ban also does not keep Inf/NaN out of vector math. Finite
> > > coordinates still overflow. For instance: magnitude = ||[FLOAT_MAX,
> > > FLOAT_MAX]|| where every coordinate is finite overflows to +Inf.
> PyTorch
> > > documents the same overflow: tensor([1e20, 1e20]).norm() is inf in
> fp32.
> > >
> > > A generated MAGNITUDE(vec) column would then be Inf while the VECTOR
> > itself
> > > was required to be finite. The ban only forbids storing specials in the
> > > coordinates, and does not prevent even simple downstream computations
> > from
> > > producing them.
> > >
> > > I suggest keeping VECTOR as the type and add an optional bool
> > > VectorType.finite (default false). Engines that want pgvector-like
> > > rejection can require the flag = true. Engines that use NumPy /
> PyTorch /
> > > Spark arrays can keep representable IEEE values. For the same reason, I
> > do
> > > not favor adding yet another type called FINITE_VECTOR.
> > >
> > > Best,
> > > Sebastian
> > >
> > >
> > > On Tue, Sep 29, 2026 at 2:03 PM Divjot Arora via dev <
> > > dev@parquet.apache.org>
> > > wrote:
> > >
> > > > I agree with moving forward with the proposal and renaming VECTOR to
> > > > FINITE_VECTOR or something
> > > > similar if others feel that better communicates its intent. I see
> value
> > > in
> > > > having a vector type that guarantees
> > > > its values are mathematically-correct vectors for a few reasons:
> > > >
> > > > 1. As others have mentioned, this is similar to the STRING logical
> type
> > > > mandating UTF-8.
> > > > 2. It is not possible to rely on statistics for detecting NaN or
> +-inf
> > > > values as all statistics are optional.
> > > > To my knowledge, there is no precedent for requiring that writers
> emit
> > > > stats for specific logical types.
> > > > 3. Without statistics, NaN/inf values could flow through the engine
> and
> > > > block them for specific functionality.
> > > > This seems to be Lance's behavior. However, I don't think this is as
> > easy
> > > > as it sounds and Lance has run
> > > > into several bugs due to this ([1] [2] for examples). Their Python
> [3]
> > > and
> > > > Rust [4] APIs default to erroring on
> > > > vectors that contain NaN values.
> > > >
> > > > I agree that Parquet is a general-purpose format, but I don't think
> > this
> > > > implies the lack of domain-specific types.
> > > > Instead, this broadens the use cases where Parquet would be
> considered
> > a
> > > > good fit. I also fully agree with
> > > > continuing work on a general fixed-size list type if we believe there
> > are
> > > > non-vector use cases that would benefit
> > > > from such guarantees at the storage layer.
> > > >
> > > > [1] https://github.com/lance-format/lance/pull/3648
> > > > [2] https://github.com/lance-format/lance/issues/8607 (see point 2
> in
> > > > "root
> > > > cause")
> > > > [3]
> > > >
> > > >
> > >
> >
> https://github.com/lancedb/lancedb/blob/0327618a62483feb6dca2336549845e5887cd48f/python/python/lancedb/table.py#L4807-L4911
> > > > [4]
> > > >
> > > >
> > >
> >
> https://github.com/lancedb/lancedb/blob/0327618a62483feb6dca2336549845e5887cd48f/rust/lancedb/src/table/add_data.rs#L105-L110
> > > >
> > > > On Mon, Sep 28, 2026 at 6:19 PM Philipp Fischbeck via dev <
> > > > dev@parquet.apache.org> wrote:
> > > >
> > > > > I also favor moving forward with the proposal as is. I see strong
> > value
> > > > and
> > > > > use cases for the numeric vector type, and I agree with the
> > > restrictions
> > > > > around NaN/Inf values and element types in general. Precedent
> exists
> > in
> > > > > other types that restrict which physical type values are logically
> > > valid
> > > > > (e.g., the geospatial types needing to be WKB-encoded).  A general
> > > > > fixed-size list also has value, but I think it makes sense to
> > separate
> > > > the
> > > > > two types. The semantics are relevant, and I can see how the two
> > types
> > > > > might diverge further in the future regarding statistics or
> > encodings.
> > > > >
> > > > > I would be okay with renaming the vector type, but it seems like
> the
> > > > VECTOR
> > > > > name is already used across other engines mentioned in this thread.
> > > > >
> > > > > Best,
> > > > > Philipp
> > > > >
> > > > > On Sat, Sep 26, 2026 at 4:37 PM Gang Wu <us...@gmail.com> wrote:
> > > > >
> > > > > > To separate orthogonal concerns, I feel we should treat the
> general
> > > > > > FixedSizeList as the underlying physical layout, and separate it
> > from
> > > > the
> > > > > > discussion of a specialized VECTOR (or broader ML tensor) logical
> > > type.
> > > > > >
> > > > > > Currently, the immediate need is to define a specialized VECTOR
> > type
> > > so
> > > > > > that different engines and formats have a common semantic
> contract
> > > for
> > > > > > interoperability.
> > > > > >
> > > > > > If the main point of contention is just the NaN/Inf/null
> > limitations,
> > > > > maybe
> > > > > > we can resolve this by adding a qualification to the name (like
> > > Andrew
> > > > > > suggested, e.g., VECTOR_EMBEDDING or FINITE_VECTOR) to avoid
> > > confusion
> > > > > with
> > > > > > broader ML use cases. Alternatively, we could introduce a boolean
> > > flag
> > > > > > (e.g., optional bool VectorType.finite) as proposed earlier.
> > > > > >
> > > > > > On Fri, Sep 25, 2026 at 3:52 AM Andrew Lamb <
> > andrewlamb11@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > I am a little confused about the points of disagreement here.
> > > > > > >
> > > > > > > Is it that the name `VECTOR` is too general for the specific
> use
> > > case
> > > > > of
> > > > > > > fixed length arrays with non-finite values used for vector
> > > > embeddings?
> > > > > > If
> > > > > > > so, maybe we could add a qualification to the name
> > (FINITE_VECTOR?
> > > > > > > FEATURE_VECTOR? VECTOR_EMBEDDING?)
> > > > > > >
> > > > > > > It seems like there is precedent for defining logical types
> that
> > > > > specify
> > > > > > > certain physical types and restrict their contents to help
> > readers
> > > > > > > interpret meaning, for example string[1] or JSON;  Defining one
> > for
> > > > > > vector
> > > > > > > embeddings seems reasonable enough to me.
> > > > > > >
> > > > > > > Andrew
> > > > > > >
> > > > > > >
> > > > > > > [1]:
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#string
> > > > > > >
> > > > > > > On Thu, Sep 24, 2026 at 3:34 PM Daniel Weeks <
> dweeks@apache.org>
> > > > > wrote:
> > > > > > >
> > > > > > > > >
> > > > > > > > > Well, Parquet is not a single-purpose format.
> > > > > > > > >
> > > > > > > > > Vector stores can choose whatever policy fits their usage,
> > but
> > > > > we're
> > > > > > > > > talking about a general-purpose file format that aims to be
> > > > broadly
> > > > > > > > > applicable.
> > > > > > > >
> > > > > > > >
> > > > > > > > I agree that the format is intended to be general purpose,
> but
> > > > types
> > > > > > > within
> > > > > > > > the format define specific representation and semantics to
> > > achieve
> > > > > that
> > > > > > > > goal.
> > > > > > > >
> > > > > > > >
> > > > > > > > > "Vector" has a more general sense than ML and it's quite
> > > trivial
> > > > to
> > > > > > > > > produce vectors with non-finite values using e.g. NumPy or
> > > Pandas
> > > > > (or
> > > > > > > > > any other piece of dataframe or scientific computing
> > software).
> > > > > > > >
> > > > > > > >
> > > > > > > >  The justification for restricting non-finite values is
> > directly
> > > > > > related
> > > > > > > to
> > > > > > > > the semantic meaning of a vector.  Vectors have specific
> > > > > > characteristics
> > > > > > > > (dimension, magnitude, direction, etc).  Non-finite values
> are
> > > > > invalid
> > > > > > > and
> > > > > > > > break those characteristics.  I'd liken it to storing
> non-utf8
> > > > values
> > > > > > in
> > > > > > > a
> > > > > > > > string and still calling it a string.
> > > > > > > >
> > > > > > > > That also differentiates a vector from a fixed length list,
> > which
> > > > is
> > > > > > more
> > > > > > > > general and does not carry the same semantic meaning.
> > > > > > > >
> > > > > > > > I also don't think we need to bundle all similarly shaped use
> > > cases
> > > > > > into
> > > > > > > > the same solution (e.g. flxed size list).  Vector is a well
> > > > > established
> > > > > > > > data type at this point and aligns with the general purpose
> > > nature
> > > > of
> > > > > > the
> > > > > > > > format (many other formats and databases natively support
> > this).
> > > > > > > >
> > > > > > > > -Dan
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Thu, Sep 24, 2026 at 11:55 AM Antoine Pitrou <
> > > > antoine@python.org>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > >
> > > > > > > > > Le 24/09/2026 à 20:11, Russell Spitzer a écrit :
> > > > > > > > > > I think something I haven't seen yet is the clear
> use-case
> > > for
> > > > > > > > non-finite
> > > > > > > > > > values within a vector type. There has been a lot of
> > > > theoretical
> > > > > > > > > discussion
> > > > > > > > > > here about use cases but what toolings currently produce
> > > these
> > > > > > > vectors
> > > > > > > > > and
> > > > > > > > > > how do they function in downstream engines? I'd like to
> > limit
> > > > > > > > introducing
> > > > > > > > > > types that have a wide range of values which are simply
> are
> > > > > failure
> > > > > > > > modes
> > > > > > > > > > of different flavors
> > > > > > > > >
> > > > > > > > > "Vector" has a more general sense than ML and it's quite
> > > trivial
> > > > to
> > > > > > > > > produce vectors with non-finite values using e.g. NumPy or
> > > Pandas
> > > > > (or
> > > > > > > > > any other piece of dataframe or scientific computing
> > software).
> > > > > > > > >
> > > > > > > > > It's not like the general concept of a vector is a IETF
> > > standard
> > > > or
> > > > > > > > > anything, so analogies with JSON or UTF8 (where the values
> > are
> > > > > > expected
> > > > > > > > > to be valid according to the respective specs) do not
> apply.
> > > > > > > > >
> > > > > > > > > This discussion would IMHO have a better chance of
> converging
> > > if
> > > > we
> > > > > > > > > separated the two orthogonal concerns of:
> > > > > > > > >
> > > > > > > > > 1. representing fixed-length list/vector data in Parquet
> > (with
> > > a
> > > > > > > > > potential limitation to non-nested types)
> > > > > > > > >
> > > > > > > > > 2. representing the information that a given column (or
> > column
> > > > > chunk,
> > > > > > > > > etc.) does not contain non-finite values
> > > > > > > > >
> > > > > > > > > ... both of which are useful on their own right, but do not
> > > have
> > > > to
> > > > > > be
> > > > > > > > > conflated and bundled together.
> > > > > > > > >
> > > > > > > > > Regards
> > > > > > > > >
> > > > > > > > > Antoine.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] Numeric Vector type: physical representation (outcome of focused call)

[Philipp Fischbeck via dev <de...@parquet.apache.org>]

Thinking about this more, I am also fine with pulling out the “NaN/Inf”
requirement. Either into a type boolean (my preference), or putting no
restrictions on the Parquet level, allowing all IEEE 754 float numbers. If
some top-level application wants to put additional constraints on the data
type, like Iceberg not allowing NaN/infinity, it can still do so one layer
above. It seems like this was the biggest point of discussion remaining.
All these options work for me, and we should move forward with this
proposal.

Concerning additional element types like bf16, they can be introduced as
Parquet types (at least in the current LIST-based spec proposal).

Concerning Will’s suggestion around representing VECTOR as a variable
length list with a MIN/MAX_LENGTH metadata restriction: I think that a
dedicated logical VECTOR type still provides enough benefits by describing
the semantics of the data, and allowing to capture future changes around
statistics. In that case, the length is better captured as an exact length
for this type specifically. I’m also not sure about other use cases for
MIN_LENGTH, so introducing a new type constraint system just for the
MIN_LENGTH=MAX_LENGTH case for vectors doesn’t have enough value in my view.

Best,
Philipp


On Thu, Oct 1, 2026 at 9:00 AM Will Edwards via dev <de...@parquet.apache.org>
wrote:

> +1 Sebastian, a vector type suggests usage beyond normalized embeddings and
> engines really aren't going to be able to elide all the checks anyway.
>
> Antoine, you suggested we explore metadata solutions too :)
>
> Today, a database supporting CHAR(N) or VARCHAR(n) can't express this in
> Parquet.  Engines can write opaque blobs they understand in the metadata
> (e.g. Arrow does) so the extra type info round-trips, but this isn't
> standardized, portable, nor interoperable.
>
> So perhaps standardizing constraint metadata or new Thrift fields (my pref)
> can generically support VECTOR(N) whilst also being more broadly useful for
> other use cases and data types?
>
> Imagine metadata or Thrift fields to tag the MIN_LENGTH and MAX_LENGTH for
> Parquet's variable-length physcial types.  These would count logical units
> (e.g. UTF8 code points for STRING) rather than bytes.
>
> Then one engine's VECTOR(N) can be written to Parquet as LIST(N) with
> MIN_LENGTH = MAX_LENGTH = N and then readers - even different engines - can
> pick up these constraints if they support doing so.  We don't actually need
> a logical type to express a VECTOR then.
>
> This also allows standard SQL's CHAR(N) and VARCHAR(N) to be expressed on
> UTF8 columns etc.
>
> This also seems generally useful in the future as we approach expressing
> other tensor data shapes and types.
>
> This won't change the mechanics: the storage type doesn't enforce these
> constraints and engines must check them. It's previously established that
> performance is just a software problem and there is no fundamental
> performance advantage to the fixed size types in Dremel-like formats like
> Parquet.
>
> Best,
> Will
>
> On Wed, 30 Sept 2026 at 16:08, Sebastian Baunsgaard <baunsgaard@apache.org
> >
> wrote:
>
> > After listening in on the last meeting Wednesday, I have been thinking
> > about this quite a bit, and thought I would share my 2 cents.
> >
> > I suggest not requiring VECTOR elements to be finite, as proposed in the
> PR
> > <https://github.com/apache/parquet-format/pull/624>.
> >
> > While Divjot and others note the specialization for UTF-8, Parquet
> already
> > defines DOUBLE and FLOAT as IEEE encodings
> > <https://github.com/apache/parquet-format/blob/master/README.md#types>.
> >
> > Making NaN or +/-Inf illegal inside VECTOR would be a new restriction on
> > those same underlying physical types. Scalar FLOAT/DOUBLE columns have
> > never had such a rule.
> >
> > It would also not match how the systems that actually produce these
> vectors
> > treat floats. Most of them use IEEE floats, e.g. Spark, DuckDB, NumPy,
> > PyTorch, TensorFlow, or lower level CUDA, oneMKL, and OpenBLAS.
> >
> > pgvector's "all elements must be finite" rule is an insert policy. While
> I
> > think Parquet as a general-purpose format should allow them out of the
> box.
> > It is a user system decision to disallow writing these values or to throw
> > errors when encountering them.
> >
> > A type-level ban also does not keep Inf/NaN out of vector math. Finite
> > coordinates still overflow. For instance: magnitude = ||[FLOAT_MAX,
> > FLOAT_MAX]|| where every coordinate is finite overflows to +Inf. PyTorch
> > documents the same overflow: tensor([1e20, 1e20]).norm() is inf in fp32.
> >
> > A generated MAGNITUDE(vec) column would then be Inf while the VECTOR
> itself
> > was required to be finite. The ban only forbids storing specials in the
> > coordinates, and does not prevent even simple downstream computations
> from
> > producing them.
> >
> > I suggest keeping VECTOR as the type and add an optional bool
> > VectorType.finite (default false). Engines that want pgvector-like
> > rejection can require the flag = true. Engines that use NumPy / PyTorch /
> > Spark arrays can keep representable IEEE values. For the same reason, I
> do
> > not favor adding yet another type called FINITE_VECTOR.
> >
> > Best,
> > Sebastian
> >
> >
> > On Tue, Sep 29, 2026 at 2:03 PM Divjot Arora via dev <
> > dev@parquet.apache.org>
> > wrote:
> >
> > > I agree with moving forward with the proposal and renaming VECTOR to
> > > FINITE_VECTOR or something
> > > similar if others feel that better communicates its intent. I see value
> > in
> > > having a vector type that guarantees
> > > its values are mathematically-correct vectors for a few reasons:
> > >
> > > 1. As others have mentioned, this is similar to the STRING logical type
> > > mandating UTF-8.
> > > 2. It is not possible to rely on statistics for detecting NaN or +-inf
> > > values as all statistics are optional.
> > > To my knowledge, there is no precedent for requiring that writers emit
> > > stats for specific logical types.
> > > 3. Without statistics, NaN/inf values could flow through the engine and
> > > block them for specific functionality.
> > > This seems to be Lance's behavior. However, I don't think this is as
> easy
> > > as it sounds and Lance has run
> > > into several bugs due to this ([1] [2] for examples). Their Python [3]
> > and
> > > Rust [4] APIs default to erroring on
> > > vectors that contain NaN values.
> > >
> > > I agree that Parquet is a general-purpose format, but I don't think
> this
> > > implies the lack of domain-specific types.
> > > Instead, this broadens the use cases where Parquet would be considered
> a
> > > good fit. I also fully agree with
> > > continuing work on a general fixed-size list type if we believe there
> are
> > > non-vector use cases that would benefit
> > > from such guarantees at the storage layer.
> > >
> > > [1] https://github.com/lance-format/lance/pull/3648
> > > [2] https://github.com/lance-format/lance/issues/8607 (see point 2 in
> > > "root
> > > cause")
> > > [3]
> > >
> > >
> >
> https://github.com/lancedb/lancedb/blob/0327618a62483feb6dca2336549845e5887cd48f/python/python/lancedb/table.py#L4807-L4911
> > > [4]
> > >
> > >
> >
> https://github.com/lancedb/lancedb/blob/0327618a62483feb6dca2336549845e5887cd48f/rust/lancedb/src/table/add_data.rs#L105-L110
> > >
> > > On Mon, Sep 28, 2026 at 6:19 PM Philipp Fischbeck via dev <
> > > dev@parquet.apache.org> wrote:
> > >
> > > > I also favor moving forward with the proposal as is. I see strong
> value
> > > and
> > > > use cases for the numeric vector type, and I agree with the
> > restrictions
> > > > around NaN/Inf values and element types in general. Precedent exists
> in
> > > > other types that restrict which physical type values are logically
> > valid
> > > > (e.g., the geospatial types needing to be WKB-encoded).  A general
> > > > fixed-size list also has value, but I think it makes sense to
> separate
> > > the
> > > > two types. The semantics are relevant, and I can see how the two
> types
> > > > might diverge further in the future regarding statistics or
> encodings.
> > > >
> > > > I would be okay with renaming the vector type, but it seems like the
> > > VECTOR
> > > > name is already used across other engines mentioned in this thread.
> > > >
> > > > Best,
> > > > Philipp
> > > >
> > > > On Sat, Sep 26, 2026 at 4:37 PM Gang Wu <us...@gmail.com> wrote:
> > > >
> > > > > To separate orthogonal concerns, I feel we should treat the general
> > > > > FixedSizeList as the underlying physical layout, and separate it
> from
> > > the
> > > > > discussion of a specialized VECTOR (or broader ML tensor) logical
> > type.
> > > > >
> > > > > Currently, the immediate need is to define a specialized VECTOR
> type
> > so
> > > > > that different engines and formats have a common semantic contract
> > for
> > > > > interoperability.
> > > > >
> > > > > If the main point of contention is just the NaN/Inf/null
> limitations,
> > > > maybe
> > > > > we can resolve this by adding a qualification to the name (like
> > Andrew
> > > > > suggested, e.g., VECTOR_EMBEDDING or FINITE_VECTOR) to avoid
> > confusion
> > > > with
> > > > > broader ML use cases. Alternatively, we could introduce a boolean
> > flag
> > > > > (e.g., optional bool VectorType.finite) as proposed earlier.
> > > > >
> > > > > On Fri, Sep 25, 2026 at 3:52 AM Andrew Lamb <
> andrewlamb11@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > I am a little confused about the points of disagreement here.
> > > > > >
> > > > > > Is it that the name `VECTOR` is too general for the specific use
> > case
> > > > of
> > > > > > fixed length arrays with non-finite values used for vector
> > > embeddings?
> > > > > If
> > > > > > so, maybe we could add a qualification to the name
> (FINITE_VECTOR?
> > > > > > FEATURE_VECTOR? VECTOR_EMBEDDING?)
> > > > > >
> > > > > > It seems like there is precedent for defining logical types that
> > > > specify
> > > > > > certain physical types and restrict their contents to help
> readers
> > > > > > interpret meaning, for example string[1] or JSON;  Defining one
> for
> > > > > vector
> > > > > > embeddings seems reasonable enough to me.
> > > > > >
> > > > > > Andrew
> > > > > >
> > > > > >
> > > > > > [1]:
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#string
> > > > > >
> > > > > > On Thu, Sep 24, 2026 at 3:34 PM Daniel Weeks <dw...@apache.org>
> > > > wrote:
> > > > > >
> > > > > > > >
> > > > > > > > Well, Parquet is not a single-purpose format.
> > > > > > > >
> > > > > > > > Vector stores can choose whatever policy fits their usage,
> but
> > > > we're
> > > > > > > > talking about a general-purpose file format that aims to be
> > > broadly
> > > > > > > > applicable.
> > > > > > >
> > > > > > >
> > > > > > > I agree that the format is intended to be general purpose, but
> > > types
> > > > > > within
> > > > > > > the format define specific representation and semantics to
> > achieve
> > > > that
> > > > > > > goal.
> > > > > > >
> > > > > > >
> > > > > > > > "Vector" has a more general sense than ML and it's quite
> > trivial
> > > to
> > > > > > > > produce vectors with non-finite values using e.g. NumPy or
> > Pandas
> > > > (or
> > > > > > > > any other piece of dataframe or scientific computing
> software).
> > > > > > >
> > > > > > >
> > > > > > >  The justification for restricting non-finite values is
> directly
> > > > > related
> > > > > > to
> > > > > > > the semantic meaning of a vector.  Vectors have specific
> > > > > characteristics
> > > > > > > (dimension, magnitude, direction, etc).  Non-finite values are
> > > > invalid
> > > > > > and
> > > > > > > break those characteristics.  I'd liken it to storing non-utf8
> > > values
> > > > > in
> > > > > > a
> > > > > > > string and still calling it a string.
> > > > > > >
> > > > > > > That also differentiates a vector from a fixed length list,
> which
> > > is
> > > > > more
> > > > > > > general and does not carry the same semantic meaning.
> > > > > > >
> > > > > > > I also don't think we need to bundle all similarly shaped use
> > cases
> > > > > into
> > > > > > > the same solution (e.g. flxed size list).  Vector is a well
> > > > established
> > > > > > > data type at this point and aligns with the general purpose
> > nature
> > > of
> > > > > the
> > > > > > > format (many other formats and databases natively support
> this).
> > > > > > >
> > > > > > > -Dan
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Thu, Sep 24, 2026 at 11:55 AM Antoine Pitrou <
> > > antoine@python.org>
> > > > > > > wrote:
> > > > > > >
> > > > > > > >
> > > > > > > > Le 24/09/2026 à 20:11, Russell Spitzer a écrit :
> > > > > > > > > I think something I haven't seen yet is the clear use-case
> > for
> > > > > > > non-finite
> > > > > > > > > values within a vector type. There has been a lot of
> > > theoretical
> > > > > > > > discussion
> > > > > > > > > here about use cases but what toolings currently produce
> > these
> > > > > > vectors
> > > > > > > > and
> > > > > > > > > how do they function in downstream engines? I'd like to
> limit
> > > > > > > introducing
> > > > > > > > > types that have a wide range of values which are simply are
> > > > failure
> > > > > > > modes
> > > > > > > > > of different flavors
> > > > > > > >
> > > > > > > > "Vector" has a more general sense than ML and it's quite
> > trivial
> > > to
> > > > > > > > produce vectors with non-finite values using e.g. NumPy or
> > Pandas
> > > > (or
> > > > > > > > any other piece of dataframe or scientific computing
> software).
> > > > > > > >
> > > > > > > > It's not like the general concept of a vector is a IETF
> > standard
> > > or
> > > > > > > > anything, so analogies with JSON or UTF8 (where the values
> are
> > > > > expected
> > > > > > > > to be valid according to the respective specs) do not apply.
> > > > > > > >
> > > > > > > > This discussion would IMHO have a better chance of converging
> > if
> > > we
> > > > > > > > separated the two orthogonal concerns of:
> > > > > > > >
> > > > > > > > 1. representing fixed-length list/vector data in Parquet
> (with
> > a
> > > > > > > > potential limitation to non-nested types)
> > > > > > > >
> > > > > > > > 2. representing the information that a given column (or
> column
> > > > chunk,
> > > > > > > > etc.) does not contain non-finite values
> > > > > > > >
> > > > > > > > ... both of which are useful on their own right, but do not
> > have
> > > to
> > > > > be
> > > > > > > > conflated and bundled together.
> > > > > > > >
> > > > > > > > Regards
> > > > > > > >
> > > > > > > > Antoine.
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] Numeric Vector type: physical representation (outcome of focused call)

[Will Edwards via dev <de...@parquet.apache.org>]

+1 Sebastian, a vector type suggests usage beyond normalized embeddings and
engines really aren't going to be able to elide all the checks anyway.

Antoine, you suggested we explore metadata solutions too :)

Today, a database supporting CHAR(N) or VARCHAR(n) can't express this in
Parquet.  Engines can write opaque blobs they understand in the metadata
(e.g. Arrow does) so the extra type info round-trips, but this isn't
standardized, portable, nor interoperable.

So perhaps standardizing constraint metadata or new Thrift fields (my pref)
can generically support VECTOR(N) whilst also being more broadly useful for
other use cases and data types?

Imagine metadata or Thrift fields to tag the MIN_LENGTH and MAX_LENGTH for
Parquet's variable-length physcial types.  These would count logical units
(e.g. UTF8 code points for STRING) rather than bytes.

Then one engine's VECTOR(N) can be written to Parquet as LIST(N) with
MIN_LENGTH = MAX_LENGTH = N and then readers - even different engines - can
pick up these constraints if they support doing so.  We don't actually need
a logical type to express a VECTOR then.

This also allows standard SQL's CHAR(N) and VARCHAR(N) to be expressed on
UTF8 columns etc.

This also seems generally useful in the future as we approach expressing
other tensor data shapes and types.

This won't change the mechanics: the storage type doesn't enforce these
constraints and engines must check them. It's previously established that
performance is just a software problem and there is no fundamental
performance advantage to the fixed size types in Dremel-like formats like
Parquet.

Best,
Will

On Wed, 30 Sept 2026 at 16:08, Sebastian Baunsgaard <ba...@apache.org>
wrote:

> After listening in on the last meeting Wednesday, I have been thinking
> about this quite a bit, and thought I would share my 2 cents.
>
> I suggest not requiring VECTOR elements to be finite, as proposed in the PR
> <https://github.com/apache/parquet-format/pull/624>.
>
> While Divjot and others note the specialization for UTF-8, Parquet already
> defines DOUBLE and FLOAT as IEEE encodings
> <https://github.com/apache/parquet-format/blob/master/README.md#types>.
>
> Making NaN or +/-Inf illegal inside VECTOR would be a new restriction on
> those same underlying physical types. Scalar FLOAT/DOUBLE columns have
> never had such a rule.
>
> It would also not match how the systems that actually produce these vectors
> treat floats. Most of them use IEEE floats, e.g. Spark, DuckDB, NumPy,
> PyTorch, TensorFlow, or lower level CUDA, oneMKL, and OpenBLAS.
>
> pgvector's "all elements must be finite" rule is an insert policy. While I
> think Parquet as a general-purpose format should allow them out of the box.
> It is a user system decision to disallow writing these values or to throw
> errors when encountering them.
>
> A type-level ban also does not keep Inf/NaN out of vector math. Finite
> coordinates still overflow. For instance: magnitude = ||[FLOAT_MAX,
> FLOAT_MAX]|| where every coordinate is finite overflows to +Inf. PyTorch
> documents the same overflow: tensor([1e20, 1e20]).norm() is inf in fp32.
>
> A generated MAGNITUDE(vec) column would then be Inf while the VECTOR itself
> was required to be finite. The ban only forbids storing specials in the
> coordinates, and does not prevent even simple downstream computations from
> producing them.
>
> I suggest keeping VECTOR as the type and add an optional bool
> VectorType.finite (default false). Engines that want pgvector-like
> rejection can require the flag = true. Engines that use NumPy / PyTorch /
> Spark arrays can keep representable IEEE values. For the same reason, I do
> not favor adding yet another type called FINITE_VECTOR.
>
> Best,
> Sebastian
>
>
> On Tue, Sep 29, 2026 at 2:03 PM Divjot Arora via dev <
> dev@parquet.apache.org>
> wrote:
>
> > I agree with moving forward with the proposal and renaming VECTOR to
> > FINITE_VECTOR or something
> > similar if others feel that better communicates its intent. I see value
> in
> > having a vector type that guarantees
> > its values are mathematically-correct vectors for a few reasons:
> >
> > 1. As others have mentioned, this is similar to the STRING logical type
> > mandating UTF-8.
> > 2. It is not possible to rely on statistics for detecting NaN or +-inf
> > values as all statistics are optional.
> > To my knowledge, there is no precedent for requiring that writers emit
> > stats for specific logical types.
> > 3. Without statistics, NaN/inf values could flow through the engine and
> > block them for specific functionality.
> > This seems to be Lance's behavior. However, I don't think this is as easy
> > as it sounds and Lance has run
> > into several bugs due to this ([1] [2] for examples). Their Python [3]
> and
> > Rust [4] APIs default to erroring on
> > vectors that contain NaN values.
> >
> > I agree that Parquet is a general-purpose format, but I don't think this
> > implies the lack of domain-specific types.
> > Instead, this broadens the use cases where Parquet would be considered a
> > good fit. I also fully agree with
> > continuing work on a general fixed-size list type if we believe there are
> > non-vector use cases that would benefit
> > from such guarantees at the storage layer.
> >
> > [1] https://github.com/lance-format/lance/pull/3648
> > [2] https://github.com/lance-format/lance/issues/8607 (see point 2 in
> > "root
> > cause")
> > [3]
> >
> >
> https://github.com/lancedb/lancedb/blob/0327618a62483feb6dca2336549845e5887cd48f/python/python/lancedb/table.py#L4807-L4911
> > [4]
> >
> >
> https://github.com/lancedb/lancedb/blob/0327618a62483feb6dca2336549845e5887cd48f/rust/lancedb/src/table/add_data.rs#L105-L110
> >
> > On Mon, Sep 28, 2026 at 6:19 PM Philipp Fischbeck via dev <
> > dev@parquet.apache.org> wrote:
> >
> > > I also favor moving forward with the proposal as is. I see strong value
> > and
> > > use cases for the numeric vector type, and I agree with the
> restrictions
> > > around NaN/Inf values and element types in general. Precedent exists in
> > > other types that restrict which physical type values are logically
> valid
> > > (e.g., the geospatial types needing to be WKB-encoded).  A general
> > > fixed-size list also has value, but I think it makes sense to separate
> > the
> > > two types. The semantics are relevant, and I can see how the two types
> > > might diverge further in the future regarding statistics or encodings.
> > >
> > > I would be okay with renaming the vector type, but it seems like the
> > VECTOR
> > > name is already used across other engines mentioned in this thread.
> > >
> > > Best,
> > > Philipp
> > >
> > > On Sat, Sep 26, 2026 at 4:37 PM Gang Wu <us...@gmail.com> wrote:
> > >
> > > > To separate orthogonal concerns, I feel we should treat the general
> > > > FixedSizeList as the underlying physical layout, and separate it from
> > the
> > > > discussion of a specialized VECTOR (or broader ML tensor) logical
> type.
> > > >
> > > > Currently, the immediate need is to define a specialized VECTOR type
> so
> > > > that different engines and formats have a common semantic contract
> for
> > > > interoperability.
> > > >
> > > > If the main point of contention is just the NaN/Inf/null limitations,
> > > maybe
> > > > we can resolve this by adding a qualification to the name (like
> Andrew
> > > > suggested, e.g., VECTOR_EMBEDDING or FINITE_VECTOR) to avoid
> confusion
> > > with
> > > > broader ML use cases. Alternatively, we could introduce a boolean
> flag
> > > > (e.g., optional bool VectorType.finite) as proposed earlier.
> > > >
> > > > On Fri, Sep 25, 2026 at 3:52 AM Andrew Lamb <an...@gmail.com>
> > > > wrote:
> > > >
> > > > > I am a little confused about the points of disagreement here.
> > > > >
> > > > > Is it that the name `VECTOR` is too general for the specific use
> case
> > > of
> > > > > fixed length arrays with non-finite values used for vector
> > embeddings?
> > > > If
> > > > > so, maybe we could add a qualification to the name (FINITE_VECTOR?
> > > > > FEATURE_VECTOR? VECTOR_EMBEDDING?)
> > > > >
> > > > > It seems like there is precedent for defining logical types that
> > > specify
> > > > > certain physical types and restrict their contents to help readers
> > > > > interpret meaning, for example string[1] or JSON;  Defining one for
> > > > vector
> > > > > embeddings seems reasonable enough to me.
> > > > >
> > > > > Andrew
> > > > >
> > > > >
> > > > > [1]:
> > > > >
> > > >
> > >
> >
> https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#string
> > > > >
> > > > > On Thu, Sep 24, 2026 at 3:34 PM Daniel Weeks <dw...@apache.org>
> > > wrote:
> > > > >
> > > > > > >
> > > > > > > Well, Parquet is not a single-purpose format.
> > > > > > >
> > > > > > > Vector stores can choose whatever policy fits their usage, but
> > > we're
> > > > > > > talking about a general-purpose file format that aims to be
> > broadly
> > > > > > > applicable.
> > > > > >
> > > > > >
> > > > > > I agree that the format is intended to be general purpose, but
> > types
> > > > > within
> > > > > > the format define specific representation and semantics to
> achieve
> > > that
> > > > > > goal.
> > > > > >
> > > > > >
> > > > > > > "Vector" has a more general sense than ML and it's quite
> trivial
> > to
> > > > > > > produce vectors with non-finite values using e.g. NumPy or
> Pandas
> > > (or
> > > > > > > any other piece of dataframe or scientific computing software).
> > > > > >
> > > > > >
> > > > > >  The justification for restricting non-finite values is directly
> > > > related
> > > > > to
> > > > > > the semantic meaning of a vector.  Vectors have specific
> > > > characteristics
> > > > > > (dimension, magnitude, direction, etc).  Non-finite values are
> > > invalid
> > > > > and
> > > > > > break those characteristics.  I'd liken it to storing non-utf8
> > values
> > > > in
> > > > > a
> > > > > > string and still calling it a string.
> > > > > >
> > > > > > That also differentiates a vector from a fixed length list, which
> > is
> > > > more
> > > > > > general and does not carry the same semantic meaning.
> > > > > >
> > > > > > I also don't think we need to bundle all similarly shaped use
> cases
> > > > into
> > > > > > the same solution (e.g. flxed size list).  Vector is a well
> > > established
> > > > > > data type at this point and aligns with the general purpose
> nature
> > of
> > > > the
> > > > > > format (many other formats and databases natively support this).
> > > > > >
> > > > > > -Dan
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Thu, Sep 24, 2026 at 11:55 AM Antoine Pitrou <
> > antoine@python.org>
> > > > > > wrote:
> > > > > >
> > > > > > >
> > > > > > > Le 24/09/2026 à 20:11, Russell Spitzer a écrit :
> > > > > > > > I think something I haven't seen yet is the clear use-case
> for
> > > > > > non-finite
> > > > > > > > values within a vector type. There has been a lot of
> > theoretical
> > > > > > > discussion
> > > > > > > > here about use cases but what toolings currently produce
> these
> > > > > vectors
> > > > > > > and
> > > > > > > > how do they function in downstream engines? I'd like to limit
> > > > > > introducing
> > > > > > > > types that have a wide range of values which are simply are
> > > failure
> > > > > > modes
> > > > > > > > of different flavors
> > > > > > >
> > > > > > > "Vector" has a more general sense than ML and it's quite
> trivial
> > to
> > > > > > > produce vectors with non-finite values using e.g. NumPy or
> Pandas
> > > (or
> > > > > > > any other piece of dataframe or scientific computing software).
> > > > > > >
> > > > > > > It's not like the general concept of a vector is a IETF
> standard
> > or
> > > > > > > anything, so analogies with JSON or UTF8 (where the values are
> > > > expected
> > > > > > > to be valid according to the respective specs) do not apply.
> > > > > > >
> > > > > > > This discussion would IMHO have a better chance of converging
> if
> > we
> > > > > > > separated the two orthogonal concerns of:
> > > > > > >
> > > > > > > 1. representing fixed-length list/vector data in Parquet (with
> a
> > > > > > > potential limitation to non-nested types)
> > > > > > >
> > > > > > > 2. representing the information that a given column (or column
> > > chunk,
> > > > > > > etc.) does not contain non-finite values
> > > > > > >
> > > > > > > ... both of which are useful on their own right, but do not
> have
> > to
> > > > be
> > > > > > > conflated and bundled together.
> > > > > > >
> > > > > > > Regards
> > > > > > >
> > > > > > > Antoine.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] Numeric Vector type: physical representation (outcome of focused call)

[Sebastian Baunsgaard <ba...@apache.org>]

After listening in on the last meeting Wednesday, I have been thinking
about this quite a bit, and thought I would share my 2 cents.

I suggest not requiring VECTOR elements to be finite, as proposed in the PR
<https://github.com/apache/parquet-format/pull/624>.

While Divjot and others note the specialization for UTF-8, Parquet already
defines DOUBLE and FLOAT as IEEE encodings
<https://github.com/apache/parquet-format/blob/master/README.md#types>.

Making NaN or +/-Inf illegal inside VECTOR would be a new restriction on
those same underlying physical types. Scalar FLOAT/DOUBLE columns have
never had such a rule.

It would also not match how the systems that actually produce these vectors
treat floats. Most of them use IEEE floats, e.g. Spark, DuckDB, NumPy,
PyTorch, TensorFlow, or lower level CUDA, oneMKL, and OpenBLAS.

pgvector's "all elements must be finite" rule is an insert policy. While I
think Parquet as a general-purpose format should allow them out of the box.
It is a user system decision to disallow writing these values or to throw
errors when encountering them.

A type-level ban also does not keep Inf/NaN out of vector math. Finite
coordinates still overflow. For instance: magnitude = ||[FLOAT_MAX,
FLOAT_MAX]|| where every coordinate is finite overflows to +Inf. PyTorch
documents the same overflow: tensor([1e20, 1e20]).norm() is inf in fp32.

A generated MAGNITUDE(vec) column would then be Inf while the VECTOR itself
was required to be finite. The ban only forbids storing specials in the
coordinates, and does not prevent even simple downstream computations from
producing them.

I suggest keeping VECTOR as the type and add an optional bool
VectorType.finite (default false). Engines that want pgvector-like
rejection can require the flag = true. Engines that use NumPy / PyTorch /
Spark arrays can keep representable IEEE values. For the same reason, I do
not favor adding yet another type called FINITE_VECTOR.

Best,
Sebastian


On Tue, Sep 29, 2026 at 2:03 PM Divjot Arora via dev <de...@parquet.apache.org>
wrote:

> I agree with moving forward with the proposal and renaming VECTOR to
> FINITE_VECTOR or something
> similar if others feel that better communicates its intent. I see value in
> having a vector type that guarantees
> its values are mathematically-correct vectors for a few reasons:
>
> 1. As others have mentioned, this is similar to the STRING logical type
> mandating UTF-8.
> 2. It is not possible to rely on statistics for detecting NaN or +-inf
> values as all statistics are optional.
> To my knowledge, there is no precedent for requiring that writers emit
> stats for specific logical types.
> 3. Without statistics, NaN/inf values could flow through the engine and
> block them for specific functionality.
> This seems to be Lance's behavior. However, I don't think this is as easy
> as it sounds and Lance has run
> into several bugs due to this ([1] [2] for examples). Their Python [3] and
> Rust [4] APIs default to erroring on
> vectors that contain NaN values.
>
> I agree that Parquet is a general-purpose format, but I don't think this
> implies the lack of domain-specific types.
> Instead, this broadens the use cases where Parquet would be considered a
> good fit. I also fully agree with
> continuing work on a general fixed-size list type if we believe there are
> non-vector use cases that would benefit
> from such guarantees at the storage layer.
>
> [1] https://github.com/lance-format/lance/pull/3648
> [2] https://github.com/lance-format/lance/issues/8607 (see point 2 in
> "root
> cause")
> [3]
>
> https://github.com/lancedb/lancedb/blob/0327618a62483feb6dca2336549845e5887cd48f/python/python/lancedb/table.py#L4807-L4911
> [4]
>
> https://github.com/lancedb/lancedb/blob/0327618a62483feb6dca2336549845e5887cd48f/rust/lancedb/src/table/add_data.rs#L105-L110
>
> On Mon, Sep 28, 2026 at 6:19 PM Philipp Fischbeck via dev <
> dev@parquet.apache.org> wrote:
>
> > I also favor moving forward with the proposal as is. I see strong value
> and
> > use cases for the numeric vector type, and I agree with the restrictions
> > around NaN/Inf values and element types in general. Precedent exists in
> > other types that restrict which physical type values are logically valid
> > (e.g., the geospatial types needing to be WKB-encoded).  A general
> > fixed-size list also has value, but I think it makes sense to separate
> the
> > two types. The semantics are relevant, and I can see how the two types
> > might diverge further in the future regarding statistics or encodings.
> >
> > I would be okay with renaming the vector type, but it seems like the
> VECTOR
> > name is already used across other engines mentioned in this thread.
> >
> > Best,
> > Philipp
> >
> > On Sat, Sep 26, 2026 at 4:37 PM Gang Wu <us...@gmail.com> wrote:
> >
> > > To separate orthogonal concerns, I feel we should treat the general
> > > FixedSizeList as the underlying physical layout, and separate it from
> the
> > > discussion of a specialized VECTOR (or broader ML tensor) logical type.
> > >
> > > Currently, the immediate need is to define a specialized VECTOR type so
> > > that different engines and formats have a common semantic contract for
> > > interoperability.
> > >
> > > If the main point of contention is just the NaN/Inf/null limitations,
> > maybe
> > > we can resolve this by adding a qualification to the name (like Andrew
> > > suggested, e.g., VECTOR_EMBEDDING or FINITE_VECTOR) to avoid confusion
> > with
> > > broader ML use cases. Alternatively, we could introduce a boolean flag
> > > (e.g., optional bool VectorType.finite) as proposed earlier.
> > >
> > > On Fri, Sep 25, 2026 at 3:52 AM Andrew Lamb <an...@gmail.com>
> > > wrote:
> > >
> > > > I am a little confused about the points of disagreement here.
> > > >
> > > > Is it that the name `VECTOR` is too general for the specific use case
> > of
> > > > fixed length arrays with non-finite values used for vector
> embeddings?
> > > If
> > > > so, maybe we could add a qualification to the name (FINITE_VECTOR?
> > > > FEATURE_VECTOR? VECTOR_EMBEDDING?)
> > > >
> > > > It seems like there is precedent for defining logical types that
> > specify
> > > > certain physical types and restrict their contents to help readers
> > > > interpret meaning, for example string[1] or JSON;  Defining one for
> > > vector
> > > > embeddings seems reasonable enough to me.
> > > >
> > > > Andrew
> > > >
> > > >
> > > > [1]:
> > > >
> > >
> >
> https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#string
> > > >
> > > > On Thu, Sep 24, 2026 at 3:34 PM Daniel Weeks <dw...@apache.org>
> > wrote:
> > > >
> > > > > >
> > > > > > Well, Parquet is not a single-purpose format.
> > > > > >
> > > > > > Vector stores can choose whatever policy fits their usage, but
> > we're
> > > > > > talking about a general-purpose file format that aims to be
> broadly
> > > > > > applicable.
> > > > >
> > > > >
> > > > > I agree that the format is intended to be general purpose, but
> types
> > > > within
> > > > > the format define specific representation and semantics to achieve
> > that
> > > > > goal.
> > > > >
> > > > >
> > > > > > "Vector" has a more general sense than ML and it's quite trivial
> to
> > > > > > produce vectors with non-finite values using e.g. NumPy or Pandas
> > (or
> > > > > > any other piece of dataframe or scientific computing software).
> > > > >
> > > > >
> > > > >  The justification for restricting non-finite values is directly
> > > related
> > > > to
> > > > > the semantic meaning of a vector.  Vectors have specific
> > > characteristics
> > > > > (dimension, magnitude, direction, etc).  Non-finite values are
> > invalid
> > > > and
> > > > > break those characteristics.  I'd liken it to storing non-utf8
> values
> > > in
> > > > a
> > > > > string and still calling it a string.
> > > > >
> > > > > That also differentiates a vector from a fixed length list, which
> is
> > > more
> > > > > general and does not carry the same semantic meaning.
> > > > >
> > > > > I also don't think we need to bundle all similarly shaped use cases
> > > into
> > > > > the same solution (e.g. flxed size list).  Vector is a well
> > established
> > > > > data type at this point and aligns with the general purpose nature
> of
> > > the
> > > > > format (many other formats and databases natively support this).
> > > > >
> > > > > -Dan
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Thu, Sep 24, 2026 at 11:55 AM Antoine Pitrou <
> antoine@python.org>
> > > > > wrote:
> > > > >
> > > > > >
> > > > > > Le 24/09/2026 à 20:11, Russell Spitzer a écrit :
> > > > > > > I think something I haven't seen yet is the clear use-case for
> > > > > non-finite
> > > > > > > values within a vector type. There has been a lot of
> theoretical
> > > > > > discussion
> > > > > > > here about use cases but what toolings currently produce these
> > > > vectors
> > > > > > and
> > > > > > > how do they function in downstream engines? I'd like to limit
> > > > > introducing
> > > > > > > types that have a wide range of values which are simply are
> > failure
> > > > > modes
> > > > > > > of different flavors
> > > > > >
> > > > > > "Vector" has a more general sense than ML and it's quite trivial
> to
> > > > > > produce vectors with non-finite values using e.g. NumPy or Pandas
> > (or
> > > > > > any other piece of dataframe or scientific computing software).
> > > > > >
> > > > > > It's not like the general concept of a vector is a IETF standard
> or
> > > > > > anything, so analogies with JSON or UTF8 (where the values are
> > > expected
> > > > > > to be valid according to the respective specs) do not apply.
> > > > > >
> > > > > > This discussion would IMHO have a better chance of converging if
> we
> > > > > > separated the two orthogonal concerns of:
> > > > > >
> > > > > > 1. representing fixed-length list/vector data in Parquet (with a
> > > > > > potential limitation to non-nested types)
> > > > > >
> > > > > > 2. representing the information that a given column (or column
> > chunk,
> > > > > > etc.) does not contain non-finite values
> > > > > >
> > > > > > ... both of which are useful on their own right, but do not have
> to
> > > be
> > > > > > conflated and bundled together.
> > > > > >
> > > > > > Regards
> > > > > >
> > > > > > Antoine.
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] Numeric Vector type: physical representation (outcome of focused call)

[Divjot Arora via dev <de...@parquet.apache.org>]

I agree with moving forward with the proposal and renaming VECTOR to
FINITE_VECTOR or something
similar if others feel that better communicates its intent. I see value in
having a vector type that guarantees
its values are mathematically-correct vectors for a few reasons:

1. As others have mentioned, this is similar to the STRING logical type
mandating UTF-8.
2. It is not possible to rely on statistics for detecting NaN or +-inf
values as all statistics are optional.
To my knowledge, there is no precedent for requiring that writers emit
stats for specific logical types.
3. Without statistics, NaN/inf values could flow through the engine and
block them for specific functionality.
This seems to be Lance's behavior. However, I don't think this is as easy
as it sounds and Lance has run
into several bugs due to this ([1] [2] for examples). Their Python [3] and
Rust [4] APIs default to erroring on
vectors that contain NaN values.

I agree that Parquet is a general-purpose format, but I don't think this
implies the lack of domain-specific types.
Instead, this broadens the use cases where Parquet would be considered a
good fit. I also fully agree with
continuing work on a general fixed-size list type if we believe there are
non-vector use cases that would benefit
from such guarantees at the storage layer.

[1] https://github.com/lance-format/lance/pull/3648
[2] https://github.com/lance-format/lance/issues/8607 (see point 2 in "root
cause")
[3]
https://github.com/lancedb/lancedb/blob/0327618a62483feb6dca2336549845e5887cd48f/python/python/lancedb/table.py#L4807-L4911
[4]
https://github.com/lancedb/lancedb/blob/0327618a62483feb6dca2336549845e5887cd48f/rust/lancedb/src/table/add_data.rs#L105-L110

On Mon, Sep 28, 2026 at 6:19 PM Philipp Fischbeck via dev <
dev@parquet.apache.org> wrote:

> I also favor moving forward with the proposal as is. I see strong value and
> use cases for the numeric vector type, and I agree with the restrictions
> around NaN/Inf values and element types in general. Precedent exists in
> other types that restrict which physical type values are logically valid
> (e.g., the geospatial types needing to be WKB-encoded).  A general
> fixed-size list also has value, but I think it makes sense to separate the
> two types. The semantics are relevant, and I can see how the two types
> might diverge further in the future regarding statistics or encodings.
>
> I would be okay with renaming the vector type, but it seems like the VECTOR
> name is already used across other engines mentioned in this thread.
>
> Best,
> Philipp
>
> On Sat, Sep 26, 2026 at 4:37 PM Gang Wu <us...@gmail.com> wrote:
>
> > To separate orthogonal concerns, I feel we should treat the general
> > FixedSizeList as the underlying physical layout, and separate it from the
> > discussion of a specialized VECTOR (or broader ML tensor) logical type.
> >
> > Currently, the immediate need is to define a specialized VECTOR type so
> > that different engines and formats have a common semantic contract for
> > interoperability.
> >
> > If the main point of contention is just the NaN/Inf/null limitations,
> maybe
> > we can resolve this by adding a qualification to the name (like Andrew
> > suggested, e.g., VECTOR_EMBEDDING or FINITE_VECTOR) to avoid confusion
> with
> > broader ML use cases. Alternatively, we could introduce a boolean flag
> > (e.g., optional bool VectorType.finite) as proposed earlier.
> >
> > On Fri, Sep 25, 2026 at 3:52 AM Andrew Lamb <an...@gmail.com>
> > wrote:
> >
> > > I am a little confused about the points of disagreement here.
> > >
> > > Is it that the name `VECTOR` is too general for the specific use case
> of
> > > fixed length arrays with non-finite values used for vector embeddings?
> > If
> > > so, maybe we could add a qualification to the name (FINITE_VECTOR?
> > > FEATURE_VECTOR? VECTOR_EMBEDDING?)
> > >
> > > It seems like there is precedent for defining logical types that
> specify
> > > certain physical types and restrict their contents to help readers
> > > interpret meaning, for example string[1] or JSON;  Defining one for
> > vector
> > > embeddings seems reasonable enough to me.
> > >
> > > Andrew
> > >
> > >
> > > [1]:
> > >
> >
> https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#string
> > >
> > > On Thu, Sep 24, 2026 at 3:34 PM Daniel Weeks <dw...@apache.org>
> wrote:
> > >
> > > > >
> > > > > Well, Parquet is not a single-purpose format.
> > > > >
> > > > > Vector stores can choose whatever policy fits their usage, but
> we're
> > > > > talking about a general-purpose file format that aims to be broadly
> > > > > applicable.
> > > >
> > > >
> > > > I agree that the format is intended to be general purpose, but types
> > > within
> > > > the format define specific representation and semantics to achieve
> that
> > > > goal.
> > > >
> > > >
> > > > > "Vector" has a more general sense than ML and it's quite trivial to
> > > > > produce vectors with non-finite values using e.g. NumPy or Pandas
> (or
> > > > > any other piece of dataframe or scientific computing software).
> > > >
> > > >
> > > >  The justification for restricting non-finite values is directly
> > related
> > > to
> > > > the semantic meaning of a vector.  Vectors have specific
> > characteristics
> > > > (dimension, magnitude, direction, etc).  Non-finite values are
> invalid
> > > and
> > > > break those characteristics.  I'd liken it to storing non-utf8 values
> > in
> > > a
> > > > string and still calling it a string.
> > > >
> > > > That also differentiates a vector from a fixed length list, which is
> > more
> > > > general and does not carry the same semantic meaning.
> > > >
> > > > I also don't think we need to bundle all similarly shaped use cases
> > into
> > > > the same solution (e.g. flxed size list).  Vector is a well
> established
> > > > data type at this point and aligns with the general purpose nature of
> > the
> > > > format (many other formats and databases natively support this).
> > > >
> > > > -Dan
> > > >
> > > >
> > > >
> > > >
> > > > On Thu, Sep 24, 2026 at 11:55 AM Antoine Pitrou <an...@python.org>
> > > > wrote:
> > > >
> > > > >
> > > > > Le 24/09/2026 à 20:11, Russell Spitzer a écrit :
> > > > > > I think something I haven't seen yet is the clear use-case for
> > > > non-finite
> > > > > > values within a vector type. There has been a lot of theoretical
> > > > > discussion
> > > > > > here about use cases but what toolings currently produce these
> > > vectors
> > > > > and
> > > > > > how do they function in downstream engines? I'd like to limit
> > > > introducing
> > > > > > types that have a wide range of values which are simply are
> failure
> > > > modes
> > > > > > of different flavors
> > > > >
> > > > > "Vector" has a more general sense than ML and it's quite trivial to
> > > > > produce vectors with non-finite values using e.g. NumPy or Pandas
> (or
> > > > > any other piece of dataframe or scientific computing software).
> > > > >
> > > > > It's not like the general concept of a vector is a IETF standard or
> > > > > anything, so analogies with JSON or UTF8 (where the values are
> > expected
> > > > > to be valid according to the respective specs) do not apply.
> > > > >
> > > > > This discussion would IMHO have a better chance of converging if we
> > > > > separated the two orthogonal concerns of:
> > > > >
> > > > > 1. representing fixed-length list/vector data in Parquet (with a
> > > > > potential limitation to non-nested types)
> > > > >
> > > > > 2. representing the information that a given column (or column
> chunk,
> > > > > etc.) does not contain non-finite values
> > > > >
> > > > > ... both of which are useful on their own right, but do not have to
> > be
> > > > > conflated and bundled together.
> > > > >
> > > > > Regards
> > > > >
> > > > > Antoine.
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] Numeric Vector type: physical representation (outcome of focused call)

[Philipp Fischbeck via dev <de...@parquet.apache.org>]

I also favor moving forward with the proposal as is. I see strong value and
use cases for the numeric vector type, and I agree with the restrictions
around NaN/Inf values and element types in general. Precedent exists in
other types that restrict which physical type values are logically valid
(e.g., the geospatial types needing to be WKB-encoded).  A general
fixed-size list also has value, but I think it makes sense to separate the
two types. The semantics are relevant, and I can see how the two types
might diverge further in the future regarding statistics or encodings.

I would be okay with renaming the vector type, but it seems like the VECTOR
name is already used across other engines mentioned in this thread.

Best,
Philipp

On Sat, Sep 26, 2026 at 4:37 PM Gang Wu <us...@gmail.com> wrote:

> To separate orthogonal concerns, I feel we should treat the general
> FixedSizeList as the underlying physical layout, and separate it from the
> discussion of a specialized VECTOR (or broader ML tensor) logical type.
>
> Currently, the immediate need is to define a specialized VECTOR type so
> that different engines and formats have a common semantic contract for
> interoperability.
>
> If the main point of contention is just the NaN/Inf/null limitations, maybe
> we can resolve this by adding a qualification to the name (like Andrew
> suggested, e.g., VECTOR_EMBEDDING or FINITE_VECTOR) to avoid confusion with
> broader ML use cases. Alternatively, we could introduce a boolean flag
> (e.g., optional bool VectorType.finite) as proposed earlier.
>
> On Fri, Sep 25, 2026 at 3:52 AM Andrew Lamb <an...@gmail.com>
> wrote:
>
> > I am a little confused about the points of disagreement here.
> >
> > Is it that the name `VECTOR` is too general for the specific use case of
> > fixed length arrays with non-finite values used for vector embeddings?
> If
> > so, maybe we could add a qualification to the name (FINITE_VECTOR?
> > FEATURE_VECTOR? VECTOR_EMBEDDING?)
> >
> > It seems like there is precedent for defining logical types that specify
> > certain physical types and restrict their contents to help readers
> > interpret meaning, for example string[1] or JSON;  Defining one for
> vector
> > embeddings seems reasonable enough to me.
> >
> > Andrew
> >
> >
> > [1]:
> >
> https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#string
> >
> > On Thu, Sep 24, 2026 at 3:34 PM Daniel Weeks <dw...@apache.org> wrote:
> >
> > > >
> > > > Well, Parquet is not a single-purpose format.
> > > >
> > > > Vector stores can choose whatever policy fits their usage, but we're
> > > > talking about a general-purpose file format that aims to be broadly
> > > > applicable.
> > >
> > >
> > > I agree that the format is intended to be general purpose, but types
> > within
> > > the format define specific representation and semantics to achieve that
> > > goal.
> > >
> > >
> > > > "Vector" has a more general sense than ML and it's quite trivial to
> > > > produce vectors with non-finite values using e.g. NumPy or Pandas (or
> > > > any other piece of dataframe or scientific computing software).
> > >
> > >
> > >  The justification for restricting non-finite values is directly
> related
> > to
> > > the semantic meaning of a vector.  Vectors have specific
> characteristics
> > > (dimension, magnitude, direction, etc).  Non-finite values are invalid
> > and
> > > break those characteristics.  I'd liken it to storing non-utf8 values
> in
> > a
> > > string and still calling it a string.
> > >
> > > That also differentiates a vector from a fixed length list, which is
> more
> > > general and does not carry the same semantic meaning.
> > >
> > > I also don't think we need to bundle all similarly shaped use cases
> into
> > > the same solution (e.g. flxed size list).  Vector is a well established
> > > data type at this point and aligns with the general purpose nature of
> the
> > > format (many other formats and databases natively support this).
> > >
> > > -Dan
> > >
> > >
> > >
> > >
> > > On Thu, Sep 24, 2026 at 11:55 AM Antoine Pitrou <an...@python.org>
> > > wrote:
> > >
> > > >
> > > > Le 24/09/2026 à 20:11, Russell Spitzer a écrit :
> > > > > I think something I haven't seen yet is the clear use-case for
> > > non-finite
> > > > > values within a vector type. There has been a lot of theoretical
> > > > discussion
> > > > > here about use cases but what toolings currently produce these
> > vectors
> > > > and
> > > > > how do they function in downstream engines? I'd like to limit
> > > introducing
> > > > > types that have a wide range of values which are simply are failure
> > > modes
> > > > > of different flavors
> > > >
> > > > "Vector" has a more general sense than ML and it's quite trivial to
> > > > produce vectors with non-finite values using e.g. NumPy or Pandas (or
> > > > any other piece of dataframe or scientific computing software).
> > > >
> > > > It's not like the general concept of a vector is a IETF standard or
> > > > anything, so analogies with JSON or UTF8 (where the values are
> expected
> > > > to be valid according to the respective specs) do not apply.
> > > >
> > > > This discussion would IMHO have a better chance of converging if we
> > > > separated the two orthogonal concerns of:
> > > >
> > > > 1. representing fixed-length list/vector data in Parquet (with a
> > > > potential limitation to non-nested types)
> > > >
> > > > 2. representing the information that a given column (or column chunk,
> > > > etc.) does not contain non-finite values
> > > >
> > > > ... both of which are useful on their own right, but do not have to
> be
> > > > conflated and bundled together.
> > > >
> > > > Regards
> > > >
> > > > Antoine.
> > > >
> > > >
> > > >
> > >
> >
>

Re: [DISCUSS] Numeric Vector type: physical representation (outcome of focused call)

[Gang Wu <us...@gmail.com>]

To separate orthogonal concerns, I feel we should treat the general
FixedSizeList as the underlying physical layout, and separate it from the
discussion of a specialized VECTOR (or broader ML tensor) logical type.

Currently, the immediate need is to define a specialized VECTOR type so
that different engines and formats have a common semantic contract for
interoperability.

If the main point of contention is just the NaN/Inf/null limitations, maybe
we can resolve this by adding a qualification to the name (like Andrew
suggested, e.g., VECTOR_EMBEDDING or FINITE_VECTOR) to avoid confusion with
broader ML use cases. Alternatively, we could introduce a boolean flag
(e.g., optional bool VectorType.finite) as proposed earlier.

On Fri, Sep 25, 2026 at 3:52 AM Andrew Lamb <an...@gmail.com> wrote:

> I am a little confused about the points of disagreement here.
>
> Is it that the name `VECTOR` is too general for the specific use case of
> fixed length arrays with non-finite values used for vector embeddings?  If
> so, maybe we could add a qualification to the name (FINITE_VECTOR?
> FEATURE_VECTOR? VECTOR_EMBEDDING?)
>
> It seems like there is precedent for defining logical types that specify
> certain physical types and restrict their contents to help readers
> interpret meaning, for example string[1] or JSON;  Defining one for vector
> embeddings seems reasonable enough to me.
>
> Andrew
>
>
> [1]:
> https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#string
>
> On Thu, Sep 24, 2026 at 3:34 PM Daniel Weeks <dw...@apache.org> wrote:
>
> > >
> > > Well, Parquet is not a single-purpose format.
> > >
> > > Vector stores can choose whatever policy fits their usage, but we're
> > > talking about a general-purpose file format that aims to be broadly
> > > applicable.
> >
> >
> > I agree that the format is intended to be general purpose, but types
> within
> > the format define specific representation and semantics to achieve that
> > goal.
> >
> >
> > > "Vector" has a more general sense than ML and it's quite trivial to
> > > produce vectors with non-finite values using e.g. NumPy or Pandas (or
> > > any other piece of dataframe or scientific computing software).
> >
> >
> >  The justification for restricting non-finite values is directly related
> to
> > the semantic meaning of a vector.  Vectors have specific characteristics
> > (dimension, magnitude, direction, etc).  Non-finite values are invalid
> and
> > break those characteristics.  I'd liken it to storing non-utf8 values in
> a
> > string and still calling it a string.
> >
> > That also differentiates a vector from a fixed length list, which is more
> > general and does not carry the same semantic meaning.
> >
> > I also don't think we need to bundle all similarly shaped use cases into
> > the same solution (e.g. flxed size list).  Vector is a well established
> > data type at this point and aligns with the general purpose nature of the
> > format (many other formats and databases natively support this).
> >
> > -Dan
> >
> >
> >
> >
> > On Thu, Sep 24, 2026 at 11:55 AM Antoine Pitrou <an...@python.org>
> > wrote:
> >
> > >
> > > Le 24/09/2026 à 20:11, Russell Spitzer a écrit :
> > > > I think something I haven't seen yet is the clear use-case for
> > non-finite
> > > > values within a vector type. There has been a lot of theoretical
> > > discussion
> > > > here about use cases but what toolings currently produce these
> vectors
> > > and
> > > > how do they function in downstream engines? I'd like to limit
> > introducing
> > > > types that have a wide range of values which are simply are failure
> > modes
> > > > of different flavors
> > >
> > > "Vector" has a more general sense than ML and it's quite trivial to
> > > produce vectors with non-finite values using e.g. NumPy or Pandas (or
> > > any other piece of dataframe or scientific computing software).
> > >
> > > It's not like the general concept of a vector is a IETF standard or
> > > anything, so analogies with JSON or UTF8 (where the values are expected
> > > to be valid according to the respective specs) do not apply.
> > >
> > > This discussion would IMHO have a better chance of converging if we
> > > separated the two orthogonal concerns of:
> > >
> > > 1. representing fixed-length list/vector data in Parquet (with a
> > > potential limitation to non-nested types)
> > >
> > > 2. representing the information that a given column (or column chunk,
> > > etc.) does not contain non-finite values
> > >
> > > ... both of which are useful on their own right, but do not have to be
> > > conflated and bundled together.
> > >
> > > Regards
> > >
> > > Antoine.
> > >
> > >
> > >
> >
>

Re: [DISCUSS] Numeric Vector type: physical representation (outcome of focused call)

[Gang Wu <us...@gmail.com>]

To separate orthogonal concerns, I feel we should treat the general
FixedSizeList as the underlying physical layout, and separate it from the
discussion of a specialized VECTOR (or broader ML tensor) logical type.

Currently, the immediate need is to define a specialized VECTOR type so
that different engines and formats have a common semantic contract for
interoperability.

If the main point of contention is just the NaN/Inf/null limitations, maybe
we can resolve this by adding a qualification to the name (like Andrew
suggested, e.g., VECTOR_EMBEDDING or FINITE_VECTOR) to avoid confusion with
broader ML use cases. Alternatively, we could introduce a boolean flag
(e.g., optional bool VectorType.finite) as proposed earlier.

On Fri, Sep 25, 2026 at 3:52 AM Andrew Lamb <an...@gmail.com> wrote:

> I am a little confused about the points of disagreement here.
>
> Is it that the name `VECTOR` is too general for the specific use case of
> fixed length arrays with non-finite values used for vector embeddings?  If
> so, maybe we could add a qualification to the name (FINITE_VECTOR?
> FEATURE_VECTOR? VECTOR_EMBEDDING?)
>
> It seems like there is precedent for defining logical types that specify
> certain physical types and restrict their contents to help readers
> interpret meaning, for example string[1] or JSON;  Defining one for vector
> embeddings seems reasonable enough to me.
>
> Andrew
>
>
> [1]:
> https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#string
>
> On Thu, Sep 24, 2026 at 3:34 PM Daniel Weeks <dw...@apache.org> wrote:
>
> > >
> > > Well, Parquet is not a single-purpose format.
> > >
> > > Vector stores can choose whatever policy fits their usage, but we're
> > > talking about a general-purpose file format that aims to be broadly
> > > applicable.
> >
> >
> > I agree that the format is intended to be general purpose, but types
> within
> > the format define specific representation and semantics to achieve that
> > goal.
> >
> >
> > > "Vector" has a more general sense than ML and it's quite trivial to
> > > produce vectors with non-finite values using e.g. NumPy or Pandas (or
> > > any other piece of dataframe or scientific computing software).
> >
> >
> >  The justification for restricting non-finite values is directly related
> to
> > the semantic meaning of a vector.  Vectors have specific characteristics
> > (dimension, magnitude, direction, etc).  Non-finite values are invalid
> and
> > break those characteristics.  I'd liken it to storing non-utf8 values in
> a
> > string and still calling it a string.
> >
> > That also differentiates a vector from a fixed length list, which is more
> > general and does not carry the same semantic meaning.
> >
> > I also don't think we need to bundle all similarly shaped use cases into
> > the same solution (e.g. flxed size list).  Vector is a well established
> > data type at this point and aligns with the general purpose nature of the
> > format (many other formats and databases natively support this).
> >
> > -Dan
> >
> >
> >
> >
> > On Thu, Sep 24, 2026 at 11:55 AM Antoine Pitrou <an...@python.org>
> > wrote:
> >
> > >
> > > Le 24/09/2026 à 20:11, Russell Spitzer a écrit :
> > > > I think something I haven't seen yet is the clear use-case for
> > non-finite
> > > > values within a vector type. There has been a lot of theoretical
> > > discussion
> > > > here about use cases but what toolings currently produce these
> vectors
> > > and
> > > > how do they function in downstream engines? I'd like to limit
> > introducing
> > > > types that have a wide range of values which are simply are failure
> > modes
> > > > of different flavors
> > >
> > > "Vector" has a more general sense than ML and it's quite trivial to
> > > produce vectors with non-finite values using e.g. NumPy or Pandas (or
> > > any other piece of dataframe or scientific computing software).
> > >
> > > It's not like the general concept of a vector is a IETF standard or
> > > anything, so analogies with JSON or UTF8 (where the values are expected
> > > to be valid according to the respective specs) do not apply.
> > >
> > > This discussion would IMHO have a better chance of converging if we
> > > separated the two orthogonal concerns of:
> > >
> > > 1. representing fixed-length list/vector data in Parquet (with a
> > > potential limitation to non-nested types)
> > >
> > > 2. representing the information that a given column (or column chunk,
> > > etc.) does not contain non-finite values
> > >
> > > ... both of which are useful on their own right, but do not have to be
> > > conflated and bundled together.
> > >
> > > Regards
> > >
> > > Antoine.
> > >
> > >
> > >
> >
>

Re: [DISCUSS] Numeric Vector type: physical representation (outcome of focused call)

[Andrew Lamb <an...@gmail.com>]

I am a little confused about the points of disagreement here.

Is it that the name `VECTOR` is too general for the specific use case of
fixed length arrays with non-finite values used for vector embeddings?  If
so, maybe we could add a qualification to the name (FINITE_VECTOR?
FEATURE_VECTOR? VECTOR_EMBEDDING?)

It seems like there is precedent for defining logical types that specify
certain physical types and restrict their contents to help readers
interpret meaning, for example string[1] or JSON;  Defining one for vector
embeddings seems reasonable enough to me.

Andrew


[1]:
https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#string

On Thu, Sep 24, 2026 at 3:34 PM Daniel Weeks <dw...@apache.org> wrote:

> >
> > Well, Parquet is not a single-purpose format.
> >
> > Vector stores can choose whatever policy fits their usage, but we're
> > talking about a general-purpose file format that aims to be broadly
> > applicable.
>
>
> I agree that the format is intended to be general purpose, but types within
> the format define specific representation and semantics to achieve that
> goal.
>
>
> > "Vector" has a more general sense than ML and it's quite trivial to
> > produce vectors with non-finite values using e.g. NumPy or Pandas (or
> > any other piece of dataframe or scientific computing software).
>
>
>  The justification for restricting non-finite values is directly related to
> the semantic meaning of a vector.  Vectors have specific characteristics
> (dimension, magnitude, direction, etc).  Non-finite values are invalid and
> break those characteristics.  I'd liken it to storing non-utf8 values in a
> string and still calling it a string.
>
> That also differentiates a vector from a fixed length list, which is more
> general and does not carry the same semantic meaning.
>
> I also don't think we need to bundle all similarly shaped use cases into
> the same solution (e.g. flxed size list).  Vector is a well established
> data type at this point and aligns with the general purpose nature of the
> format (many other formats and databases natively support this).
>
> -Dan
>
>
>
>
> On Thu, Sep 24, 2026 at 11:55 AM Antoine Pitrou <an...@python.org>
> wrote:
>
> >
> > Le 24/09/2026 à 20:11, Russell Spitzer a écrit :
> > > I think something I haven't seen yet is the clear use-case for
> non-finite
> > > values within a vector type. There has been a lot of theoretical
> > discussion
> > > here about use cases but what toolings currently produce these vectors
> > and
> > > how do they function in downstream engines? I'd like to limit
> introducing
> > > types that have a wide range of values which are simply are failure
> modes
> > > of different flavors
> >
> > "Vector" has a more general sense than ML and it's quite trivial to
> > produce vectors with non-finite values using e.g. NumPy or Pandas (or
> > any other piece of dataframe or scientific computing software).
> >
> > It's not like the general concept of a vector is a IETF standard or
> > anything, so analogies with JSON or UTF8 (where the values are expected
> > to be valid according to the respective specs) do not apply.
> >
> > This discussion would IMHO have a better chance of converging if we
> > separated the two orthogonal concerns of:
> >
> > 1. representing fixed-length list/vector data in Parquet (with a
> > potential limitation to non-nested types)
> >
> > 2. representing the information that a given column (or column chunk,
> > etc.) does not contain non-finite values
> >
> > ... both of which are useful on their own right, but do not have to be
> > conflated and bundled together.
> >
> > Regards
> >
> > Antoine.
> >
> >
> >
>

Re: [DISCUSS] Numeric Vector type: physical representation (outcome of focused call)

[Andrew Lamb <an...@gmail.com>]

I am a little confused about the points of disagreement here.

Is it that the name `VECTOR` is too general for the specific use case of
fixed length arrays with non-finite values used for vector embeddings?  If
so, maybe we could add a qualification to the name (FINITE_VECTOR?
FEATURE_VECTOR? VECTOR_EMBEDDING?)

It seems like there is precedent for defining logical types that specify
certain physical types and restrict their contents to help readers
interpret meaning, for example string[1] or JSON;  Defining one for vector
embeddings seems reasonable enough to me.

Andrew


[1]:
https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#string

On Thu, Sep 24, 2026 at 3:34 PM Daniel Weeks <dw...@apache.org> wrote:

> >
> > Well, Parquet is not a single-purpose format.
> >
> > Vector stores can choose whatever policy fits their usage, but we're
> > talking about a general-purpose file format that aims to be broadly
> > applicable.
>
>
> I agree that the format is intended to be general purpose, but types within
> the format define specific representation and semantics to achieve that
> goal.
>
>
> > "Vector" has a more general sense than ML and it's quite trivial to
> > produce vectors with non-finite values using e.g. NumPy or Pandas (or
> > any other piece of dataframe or scientific computing software).
>
>
>  The justification for restricting non-finite values is directly related to
> the semantic meaning of a vector.  Vectors have specific characteristics
> (dimension, magnitude, direction, etc).  Non-finite values are invalid and
> break those characteristics.  I'd liken it to storing non-utf8 values in a
> string and still calling it a string.
>
> That also differentiates a vector from a fixed length list, which is more
> general and does not carry the same semantic meaning.
>
> I also don't think we need to bundle all similarly shaped use cases into
> the same solution (e.g. flxed size list).  Vector is a well established
> data type at this point and aligns with the general purpose nature of the
> format (many other formats and databases natively support this).
>
> -Dan
>
>
>
>
> On Thu, Sep 24, 2026 at 11:55 AM Antoine Pitrou <an...@python.org>
> wrote:
>
> >
> > Le 24/09/2026 à 20:11, Russell Spitzer a écrit :
> > > I think something I haven't seen yet is the clear use-case for
> non-finite
> > > values within a vector type. There has been a lot of theoretical
> > discussion
> > > here about use cases but what toolings currently produce these vectors
> > and
> > > how do they function in downstream engines? I'd like to limit
> introducing
> > > types that have a wide range of values which are simply are failure
> modes
> > > of different flavors
> >
> > "Vector" has a more general sense than ML and it's quite trivial to
> > produce vectors with non-finite values using e.g. NumPy or Pandas (or
> > any other piece of dataframe or scientific computing software).
> >
> > It's not like the general concept of a vector is a IETF standard or
> > anything, so analogies with JSON or UTF8 (where the values are expected
> > to be valid according to the respective specs) do not apply.
> >
> > This discussion would IMHO have a better chance of converging if we
> > separated the two orthogonal concerns of:
> >
> > 1. representing fixed-length list/vector data in Parquet (with a
> > potential limitation to non-nested types)
> >
> > 2. representing the information that a given column (or column chunk,
> > etc.) does not contain non-finite values
> >
> > ... both of which are useful on their own right, but do not have to be
> > conflated and bundled together.
> >
> > Regards
> >
> > Antoine.
> >
> >
> >
>

Re: [DISCUSS] Numeric Vector type: physical representation (outcome of focused call)

[Daniel Weeks <dw...@apache.org>]

>
> Well, Parquet is not a single-purpose format.
>
> Vector stores can choose whatever policy fits their usage, but we're
> talking about a general-purpose file format that aims to be broadly
> applicable.


I agree that the format is intended to be general purpose, but types within
the format define specific representation and semantics to achieve that
goal.


> "Vector" has a more general sense than ML and it's quite trivial to
> produce vectors with non-finite values using e.g. NumPy or Pandas (or
> any other piece of dataframe or scientific computing software).


 The justification for restricting non-finite values is directly related to
the semantic meaning of a vector.  Vectors have specific characteristics
(dimension, magnitude, direction, etc).  Non-finite values are invalid and
break those characteristics.  I'd liken it to storing non-utf8 values in a
string and still calling it a string.

That also differentiates a vector from a fixed length list, which is more
general and does not carry the same semantic meaning.

I also don't think we need to bundle all similarly shaped use cases into
the same solution (e.g. flxed size list).  Vector is a well established
data type at this point and aligns with the general purpose nature of the
format (many other formats and databases natively support this).

-Dan




On Thu, Sep 24, 2026 at 11:55 AM Antoine Pitrou <an...@python.org> wrote:

>
> Le 24/09/2026 à 20:11, Russell Spitzer a écrit :
> > I think something I haven't seen yet is the clear use-case for non-finite
> > values within a vector type. There has been a lot of theoretical
> discussion
> > here about use cases but what toolings currently produce these vectors
> and
> > how do they function in downstream engines? I'd like to limit introducing
> > types that have a wide range of values which are simply are failure modes
> > of different flavors
>
> "Vector" has a more general sense than ML and it's quite trivial to
> produce vectors with non-finite values using e.g. NumPy or Pandas (or
> any other piece of dataframe or scientific computing software).
>
> It's not like the general concept of a vector is a IETF standard or
> anything, so analogies with JSON or UTF8 (where the values are expected
> to be valid according to the respective specs) do not apply.
>
> This discussion would IMHO have a better chance of converging if we
> separated the two orthogonal concerns of:
>
> 1. representing fixed-length list/vector data in Parquet (with a
> potential limitation to non-nested types)
>
> 2. representing the information that a given column (or column chunk,
> etc.) does not contain non-finite values
>
> ... both of which are useful on their own right, but do not have to be
> conflated and bundled together.
>
> Regards
>
> Antoine.
>
>
>

Re: [DISCUSS] Numeric Vector type: physical representation (outcome of focused call)

[Antoine Pitrou <an...@python.org>]

Le 24/09/2026 à 20:11, Russell Spitzer a écrit :
> I think something I haven't seen yet is the clear use-case for non-finite
> values within a vector type. There has been a lot of theoretical discussion
> here about use cases but what toolings currently produce these vectors and
> how do they function in downstream engines? I'd like to limit introducing
> types that have a wide range of values which are simply are failure modes
> of different flavors

"Vector" has a more general sense than ML and it's quite trivial to 
produce vectors with non-finite values using e.g. NumPy or Pandas (or 
any other piece of dataframe or scientific computing software).

It's not like the general concept of a vector is a IETF standard or 
anything, so analogies with JSON or UTF8 (where the values are expected 
to be valid according to the respective specs) do not apply.

This discussion would IMHO have a better chance of converging if we 
separated the two orthogonal concerns of:

1. representing fixed-length list/vector data in Parquet (with a 
potential limitation to non-nested types)

2. representing the information that a given column (or column chunk, 
etc.) does not contain non-finite values

... both of which are useful on their own right, but do not have to be 
conflated and bundled together.

Regards

Antoine.

Re: [DISCUSS] Numeric Vector type: physical representation (outcome of focused call)

[Russell Spitzer <ru...@gmail.com>]

I think something I haven't seen yet is the clear use-case for non-finite
values within a vector type. There has been a lot of theoretical discussion
here about use cases but what toolings currently produce these vectors and
how do they function in downstream engines? I'd like to limit introducing
types that have a wide range of values which are simply are failure modes
of different flavors

I also don't find any issue with the attitude of folks in this discussion,
it seems reasonable that people are trying to work together to get things
done in the format for the practical use cases they know about.

On Thu, Sep 24, 2026 at 8:49 AM Antoine Pitrou <an...@python.org> wrote:

> Le 24/09/2026 à 08:48, Will Edwards via dev a écrit :
> >
> > So is Parquet just growing a vector column type for the fixed-size
> > embeddings used in nearest-neighbour search to keep up with mainstream
> > databases, or is it jumping to support the broader set of ML tensor data
> to
> > accommodate all the random things an ML engineer might want to jam into a
> > Parquet file today?  I guess that is the question?
>
> Yes, that is a question and that was the heart of my position as well:
> """
> Well, Parquet is not a single-purpose format.
>
> Vector stores can choose whatever policy fits their usage, but we're
> talking about a general-purpose file format that aims to be broadly
> applicable.
> """
>
> I didn't see any objection to this point.
>
> I also didn't see any explanation why allowing non-finite values would
> actually hamper database usage. There are precedents where we don't
> impose limitations to data types which may be further limited in
> databases (for example timestamps).
>
> I also didn't see any consideration for the more logical solution of
> adding metadata to tell whether a numeric column may contain non-finite
> values (regardless of whether it's in a vector or not).
>
> And in the end I find that some people here are having a very cavalier
> attitude towards discussion and reaching consensus.
>
> Regards
>
> Antoine.
>
>
>

Re: [DISCUSS] Numeric Vector type: physical representation (outcome of focused call)

[Russell Spitzer <ru...@gmail.com>]

I think something I haven't seen yet is the clear use-case for non-finite
values within a vector type. There has been a lot of theoretical discussion
here about use cases but what toolings currently produce these vectors and
how do they function in downstream engines? I'd like to limit introducing
types that have a wide range of values which are simply are failure modes
of different flavors

I also don't find any issue with the attitude of folks in this discussion,
it seems reasonable that people are trying to work together to get things
done in the format for the practical use cases they know about.

On Thu, Sep 24, 2026 at 8:49 AM Antoine Pitrou <an...@python.org> wrote:

> Le 24/09/2026 à 08:48, Will Edwards via dev a écrit :
> >
> > So is Parquet just growing a vector column type for the fixed-size
> > embeddings used in nearest-neighbour search to keep up with mainstream
> > databases, or is it jumping to support the broader set of ML tensor data
> to
> > accommodate all the random things an ML engineer might want to jam into a
> > Parquet file today?  I guess that is the question?
>
> Yes, that is a question and that was the heart of my position as well:
> """
> Well, Parquet is not a single-purpose format.
>
> Vector stores can choose whatever policy fits their usage, but we're
> talking about a general-purpose file format that aims to be broadly
> applicable.
> """
>
> I didn't see any objection to this point.
>
> I also didn't see any explanation why allowing non-finite values would
> actually hamper database usage. There are precedents where we don't
> impose limitations to data types which may be further limited in
> databases (for example timestamps).
>
> I also didn't see any consideration for the more logical solution of
> adding metadata to tell whether a numeric column may contain non-finite
> values (regardless of whether it's in a vector or not).
>
> And in the end I find that some people here are having a very cavalier
> attitude towards discussion and reaching consensus.
>
> Regards
>
> Antoine.
>
>
>

Re: [DISCUSS] Numeric Vector type: physical representation (outcome of focused call)

[Antoine Pitrou <an...@python.org>]

Le 24/09/2026 à 08:48, Will Edwards via dev a écrit :
> 
> So is Parquet just growing a vector column type for the fixed-size
> embeddings used in nearest-neighbour search to keep up with mainstream
> databases, or is it jumping to support the broader set of ML tensor data to
> accommodate all the random things an ML engineer might want to jam into a
> Parquet file today?  I guess that is the question?

Yes, that is a question and that was the heart of my position as well:
"""
Well, Parquet is not a single-purpose format.

Vector stores can choose whatever policy fits their usage, but we're
talking about a general-purpose file format that aims to be broadly
applicable.
"""

I didn't see any objection to this point.

I also didn't see any explanation why allowing non-finite values would 
actually hamper database usage. There are precedents where we don't 
impose limitations to data types which may be further limited in 
databases (for example timestamps).

I also didn't see any consideration for the more logical solution of 
adding metadata to tell whether a numeric column may contain non-finite 
values (regardless of whether it's in a vector or not).

And in the end I find that some people here are having a very cavalier 
attitude towards discussion and reaching consensus.

Regards

Antoine.

Re: [DISCUSS] Numeric Vector type: physical representation (outcome of focused call)

[Will Edwards via dev <de...@parquet.apache.org>]

Hi all,

Initially, when I heard about formalizing the logical type for vectors, I
thought of general ML tensor data.  But after a few chats during the sync
calls, I began to realise that Databricks narrowly wants a vector type to
store fixed-length embeddings for nearest neighbour vector search, not a
general ML tensor representation?  In that light, the constraints regarding
NaN, null, and infinity make sense, as does comparing the proposal to
vector column support in other databases.

Vector search is cool.  In the early ML explosion, specialist databases
handled this, but mainstream databases later added general support,
offering the advantage of not needing multiple servers etc. It's a bit like
how mainstream databases eventually made JSON a first class column.

When you look at Arrow and Lance, etc., you see a much broader set of
features for tensor data.  They also feature error handling for vector
searches that consume vectors not meeting the constraints - a common issue
with messy, real-world ML data.

So is Parquet just growing a vector column type for the fixed-size
embeddings used in nearest-neighbour search to keep up with mainstream
databases, or is it jumping to support the broader set of ML tensor data to
accommodate all the random things an ML engineer might want to jam into a
Parquet file today?  I guess that is the question?

best,
Will


On Wed, 23 Sept 2026 at 21:47, Russell Spitzer <ru...@gmail.com>
wrote:

> Agreed with finite vectors. It’s a clearly supported type in the greater
> ecosystem. Having nonfinite vectors seems to be the exception and as Alkis
> mentioned, usually just a failure mode for engines which allow them.
>
> On Wed, Sep 23, 2026 at 11:22 AM Daniel Weeks <dw...@apache.org> wrote:
>
> > I agree with moving forward with the proposal as-is.
> >
> > I think what we see from other databases with native vector support is
> that
> > they treat them as formal vectors, which distinguishes them from just a
> > fixed-size array.
> >
> > It also shows that this is a well-supported type and not some niche type
> > that is not generally useful/accepted.
> >
> > -Dan
> >
> > On Wed, Sep 23, 2026 at 11:04 AM Alkis Evlogimenos via dev <
> > dev@parquet.apache.org> wrote:
> >
> > > Engines that disallow nulls, nans and infs:
> > > - postgress
> > > - oracle
> > > - mysql
> > > - sql server
> > > - also: milvus, qdrant, pinecone
> > >
> > > Notable dbs that allow nulls, nans and infs:
> > > - duckdb
> > > - clickhouse
> > > These seem to have this behavior because they do not have a specific
> > vector
> > > type so they use array(float) or fixed-size-array(float) and disable
> > > functionality when nulls or non-finite values are present.
> > >
> > >
> > > So VECTOR across most database engines disallow nulls, nans and infs.
> > This
> > > means better interoperability between parquet and such dbs so I think
> we
> > > should move ahead with the proposal as is.
> > >
> > >
> > > On Wed, Sep 23, 2026 at 5:00 PM Antoine Pitrou <an...@python.org>
> > wrote:
> > >
> > > >
> > > > Le 23/09/2026 à 16:26, Rok Mihevc a écrit :
> > > > >> Well, Parquet is not a single-purpose format.
> > > > >>
> > > > >> Vector stores can choose whatever policy fits their usage, but
> we're
> > > > >> talking about a general-purpose file format that aims to be
> broadly
> > > > >> applicable.
> > > > >
> > > > > What is the gate for Parquet to define something as a logical type?
> > > >
> > > > There is no "gate". But we should act in the broader community's
> > > > interest. A limitation that is only motivated by the needs of a
> single
> > > > segment of the user base goes against that goal.
> > > >
> > > > >> Why not? Why shouldn't it be used for storing, for example, NumPy
> > > tensor
> > > > >> data (which can contain NaNs and infinites)?
> > > > >
> > > > > A multidimensional arrays logical type might come with other
> > > > > requirements (shapes, strides, dimension names), that vector
> > > > > applications might not have.
> > > >
> > > > Well, first, you can use NumPy just for 1D data, and I'm sure some
> > > > people do that. Second, strides wouldn't be saved in Parquet where
> they
> > > > are meaningless. Third, all of this can be conveyed as additional
> > > > key-value metadata.
> > > >
> > > > >> It does not make sense to have *both* a "specialist" Vector type
> > and a
> > > > >> "general-purpose" FixedSizeList type. Parquet types are a
> > > > >> general-purpose vocabulary from which you can build up more
> > > specialized
> > > > >> applications. Specialist types can be left to domain-specific SW
> > users
> > > > >> of Parquet.
> > > > >>
> > > > >> (for example, Arrow implementations serialize the Arrow schema in
> > > > >> Parquet metadata so that the Arrow schema can be rebuilt at read
> > time,
> > > > >> without mandating that every Arrow datatype has a Parquet
> > equivalent)
> > > > >
> > > > > That does seem to work nicely for Arrow [0]. Should we revisit the
> > > > Parquet
> > > > > extensions discussion [1][2]?
> > > >
> > > > I think we should, and then specialized communities such as ML can
> > > > define their own specialized extension types (with arbitrary
> semantics)
> > > > on top of regular Parquet logical/physical types.
> > > >
> > > > Regards
> > > >
> > > > Antoine.
> > > >
> > > >
> > > >
> > >
> >
>

Re: [DISCUSS] Numeric Vector type: physical representation (outcome of focused call)

[Will Edwards via dev <de...@parquet.apache.org>]

Hi all,

Initially, when I heard about formalizing the logical type for vectors, I
thought of general ML tensor data.  But after a few chats during the sync
calls, I began to realise that Databricks narrowly wants a vector type to
store fixed-length embeddings for nearest neighbour vector search, not a
general ML tensor representation?  In that light, the constraints regarding
NaN, null, and infinity make sense, as does comparing the proposal to
vector column support in other databases.

Vector search is cool.  In the early ML explosion, specialist databases
handled this, but mainstream databases later added general support,
offering the advantage of not needing multiple servers etc. It's a bit like
how mainstream databases eventually made JSON a first class column.

When you look at Arrow and Lance, etc., you see a much broader set of
features for tensor data.  They also feature error handling for vector
searches that consume vectors not meeting the constraints - a common issue
with messy, real-world ML data.

So is Parquet just growing a vector column type for the fixed-size
embeddings used in nearest-neighbour search to keep up with mainstream
databases, or is it jumping to support the broader set of ML tensor data to
accommodate all the random things an ML engineer might want to jam into a
Parquet file today?  I guess that is the question?

best,
Will


On Wed, 23 Sept 2026 at 21:47, Russell Spitzer <ru...@gmail.com>
wrote:

> Agreed with finite vectors. It’s a clearly supported type in the greater
> ecosystem. Having nonfinite vectors seems to be the exception and as Alkis
> mentioned, usually just a failure mode for engines which allow them.
>
> On Wed, Sep 23, 2026 at 11:22 AM Daniel Weeks <dw...@apache.org> wrote:
>
> > I agree with moving forward with the proposal as-is.
> >
> > I think what we see from other databases with native vector support is
> that
> > they treat them as formal vectors, which distinguishes them from just a
> > fixed-size array.
> >
> > It also shows that this is a well-supported type and not some niche type
> > that is not generally useful/accepted.
> >
> > -Dan
> >
> > On Wed, Sep 23, 2026 at 11:04 AM Alkis Evlogimenos via dev <
> > dev@parquet.apache.org> wrote:
> >
> > > Engines that disallow nulls, nans and infs:
> > > - postgress
> > > - oracle
> > > - mysql
> > > - sql server
> > > - also: milvus, qdrant, pinecone
> > >
> > > Notable dbs that allow nulls, nans and infs:
> > > - duckdb
> > > - clickhouse
> > > These seem to have this behavior because they do not have a specific
> > vector
> > > type so they use array(float) or fixed-size-array(float) and disable
> > > functionality when nulls or non-finite values are present.
> > >
> > >
> > > So VECTOR across most database engines disallow nulls, nans and infs.
> > This
> > > means better interoperability between parquet and such dbs so I think
> we
> > > should move ahead with the proposal as is.
> > >
> > >
> > > On Wed, Sep 23, 2026 at 5:00 PM Antoine Pitrou <an...@python.org>
> > wrote:
> > >
> > > >
> > > > Le 23/09/2026 à 16:26, Rok Mihevc a écrit :
> > > > >> Well, Parquet is not a single-purpose format.
> > > > >>
> > > > >> Vector stores can choose whatever policy fits their usage, but
> we're
> > > > >> talking about a general-purpose file format that aims to be
> broadly
> > > > >> applicable.
> > > > >
> > > > > What is the gate for Parquet to define something as a logical type?
> > > >
> > > > There is no "gate". But we should act in the broader community's
> > > > interest. A limitation that is only motivated by the needs of a
> single
> > > > segment of the user base goes against that goal.
> > > >
> > > > >> Why not? Why shouldn't it be used for storing, for example, NumPy
> > > tensor
> > > > >> data (which can contain NaNs and infinites)?
> > > > >
> > > > > A multidimensional arrays logical type might come with other
> > > > > requirements (shapes, strides, dimension names), that vector
> > > > > applications might not have.
> > > >
> > > > Well, first, you can use NumPy just for 1D data, and I'm sure some
> > > > people do that. Second, strides wouldn't be saved in Parquet where
> they
> > > > are meaningless. Third, all of this can be conveyed as additional
> > > > key-value metadata.
> > > >
> > > > >> It does not make sense to have *both* a "specialist" Vector type
> > and a
> > > > >> "general-purpose" FixedSizeList type. Parquet types are a
> > > > >> general-purpose vocabulary from which you can build up more
> > > specialized
> > > > >> applications. Specialist types can be left to domain-specific SW
> > users
> > > > >> of Parquet.
> > > > >>
> > > > >> (for example, Arrow implementations serialize the Arrow schema in
> > > > >> Parquet metadata so that the Arrow schema can be rebuilt at read
> > time,
> > > > >> without mandating that every Arrow datatype has a Parquet
> > equivalent)
> > > > >
> > > > > That does seem to work nicely for Arrow [0]. Should we revisit the
> > > > Parquet
> > > > > extensions discussion [1][2]?
> > > >
> > > > I think we should, and then specialized communities such as ML can
> > > > define their own specialized extension types (with arbitrary
> semantics)
> > > > on top of regular Parquet logical/physical types.
> > > >
> > > > Regards
> > > >
> > > > Antoine.
> > > >
> > > >
> > > >
> > >
> >
>

Re: [DISCUSS] Numeric Vector type: physical representation (outcome of focused call)

[Russell Spitzer <ru...@gmail.com>]

Agreed with finite vectors. It’s a clearly supported type in the greater
ecosystem. Having nonfinite vectors seems to be the exception and as Alkis
mentioned, usually just a failure mode for engines which allow them.

On Wed, Sep 23, 2026 at 11:22 AM Daniel Weeks <dw...@apache.org> wrote:

> I agree with moving forward with the proposal as-is.
>
> I think what we see from other databases with native vector support is that
> they treat them as formal vectors, which distinguishes them from just a
> fixed-size array.
>
> It also shows that this is a well-supported type and not some niche type
> that is not generally useful/accepted.
>
> -Dan
>
> On Wed, Sep 23, 2026 at 11:04 AM Alkis Evlogimenos via dev <
> dev@parquet.apache.org> wrote:
>
> > Engines that disallow nulls, nans and infs:
> > - postgress
> > - oracle
> > - mysql
> > - sql server
> > - also: milvus, qdrant, pinecone
> >
> > Notable dbs that allow nulls, nans and infs:
> > - duckdb
> > - clickhouse
> > These seem to have this behavior because they do not have a specific
> vector
> > type so they use array(float) or fixed-size-array(float) and disable
> > functionality when nulls or non-finite values are present.
> >
> >
> > So VECTOR across most database engines disallow nulls, nans and infs.
> This
> > means better interoperability between parquet and such dbs so I think we
> > should move ahead with the proposal as is.
> >
> >
> > On Wed, Sep 23, 2026 at 5:00 PM Antoine Pitrou <an...@python.org>
> wrote:
> >
> > >
> > > Le 23/09/2026 à 16:26, Rok Mihevc a écrit :
> > > >> Well, Parquet is not a single-purpose format.
> > > >>
> > > >> Vector stores can choose whatever policy fits their usage, but we're
> > > >> talking about a general-purpose file format that aims to be broadly
> > > >> applicable.
> > > >
> > > > What is the gate for Parquet to define something as a logical type?
> > >
> > > There is no "gate". But we should act in the broader community's
> > > interest. A limitation that is only motivated by the needs of a single
> > > segment of the user base goes against that goal.
> > >
> > > >> Why not? Why shouldn't it be used for storing, for example, NumPy
> > tensor
> > > >> data (which can contain NaNs and infinites)?
> > > >
> > > > A multidimensional arrays logical type might come with other
> > > > requirements (shapes, strides, dimension names), that vector
> > > > applications might not have.
> > >
> > > Well, first, you can use NumPy just for 1D data, and I'm sure some
> > > people do that. Second, strides wouldn't be saved in Parquet where they
> > > are meaningless. Third, all of this can be conveyed as additional
> > > key-value metadata.
> > >
> > > >> It does not make sense to have *both* a "specialist" Vector type
> and a
> > > >> "general-purpose" FixedSizeList type. Parquet types are a
> > > >> general-purpose vocabulary from which you can build up more
> > specialized
> > > >> applications. Specialist types can be left to domain-specific SW
> users
> > > >> of Parquet.
> > > >>
> > > >> (for example, Arrow implementations serialize the Arrow schema in
> > > >> Parquet metadata so that the Arrow schema can be rebuilt at read
> time,
> > > >> without mandating that every Arrow datatype has a Parquet
> equivalent)
> > > >
> > > > That does seem to work nicely for Arrow [0]. Should we revisit the
> > > Parquet
> > > > extensions discussion [1][2]?
> > >
> > > I think we should, and then specialized communities such as ML can
> > > define their own specialized extension types (with arbitrary semantics)
> > > on top of regular Parquet logical/physical types.
> > >
> > > Regards
> > >
> > > Antoine.
> > >
> > >
> > >
> >
>

Re: [DISCUSS] Numeric Vector type: physical representation (outcome of focused call)

[Russell Spitzer <ru...@gmail.com>]

Agreed with finite vectors. It’s a clearly supported type in the greater
ecosystem. Having nonfinite vectors seems to be the exception and as Alkis
mentioned, usually just a failure mode for engines which allow them.

On Wed, Sep 23, 2026 at 11:22 AM Daniel Weeks <dw...@apache.org> wrote:

> I agree with moving forward with the proposal as-is.
>
> I think what we see from other databases with native vector support is that
> they treat them as formal vectors, which distinguishes them from just a
> fixed-size array.
>
> It also shows that this is a well-supported type and not some niche type
> that is not generally useful/accepted.
>
> -Dan
>
> On Wed, Sep 23, 2026 at 11:04 AM Alkis Evlogimenos via dev <
> dev@parquet.apache.org> wrote:
>
> > Engines that disallow nulls, nans and infs:
> > - postgress
> > - oracle
> > - mysql
> > - sql server
> > - also: milvus, qdrant, pinecone
> >
> > Notable dbs that allow nulls, nans and infs:
> > - duckdb
> > - clickhouse
> > These seem to have this behavior because they do not have a specific
> vector
> > type so they use array(float) or fixed-size-array(float) and disable
> > functionality when nulls or non-finite values are present.
> >
> >
> > So VECTOR across most database engines disallow nulls, nans and infs.
> This
> > means better interoperability between parquet and such dbs so I think we
> > should move ahead with the proposal as is.
> >
> >
> > On Wed, Sep 23, 2026 at 5:00 PM Antoine Pitrou <an...@python.org>
> wrote:
> >
> > >
> > > Le 23/09/2026 à 16:26, Rok Mihevc a écrit :
> > > >> Well, Parquet is not a single-purpose format.
> > > >>
> > > >> Vector stores can choose whatever policy fits their usage, but we're
> > > >> talking about a general-purpose file format that aims to be broadly
> > > >> applicable.
> > > >
> > > > What is the gate for Parquet to define something as a logical type?
> > >
> > > There is no "gate". But we should act in the broader community's
> > > interest. A limitation that is only motivated by the needs of a single
> > > segment of the user base goes against that goal.
> > >
> > > >> Why not? Why shouldn't it be used for storing, for example, NumPy
> > tensor
> > > >> data (which can contain NaNs and infinites)?
> > > >
> > > > A multidimensional arrays logical type might come with other
> > > > requirements (shapes, strides, dimension names), that vector
> > > > applications might not have.
> > >
> > > Well, first, you can use NumPy just for 1D data, and I'm sure some
> > > people do that. Second, strides wouldn't be saved in Parquet where they
> > > are meaningless. Third, all of this can be conveyed as additional
> > > key-value metadata.
> > >
> > > >> It does not make sense to have *both* a "specialist" Vector type
> and a
> > > >> "general-purpose" FixedSizeList type. Parquet types are a
> > > >> general-purpose vocabulary from which you can build up more
> > specialized
> > > >> applications. Specialist types can be left to domain-specific SW
> users
> > > >> of Parquet.
> > > >>
> > > >> (for example, Arrow implementations serialize the Arrow schema in
> > > >> Parquet metadata so that the Arrow schema can be rebuilt at read
> time,
> > > >> without mandating that every Arrow datatype has a Parquet
> equivalent)
> > > >
> > > > That does seem to work nicely for Arrow [0]. Should we revisit the
> > > Parquet
> > > > extensions discussion [1][2]?
> > >
> > > I think we should, and then specialized communities such as ML can
> > > define their own specialized extension types (with arbitrary semantics)
> > > on top of regular Parquet logical/physical types.
> > >
> > > Regards
> > >
> > > Antoine.
> > >
> > >
> > >
> >
>

Re: [DISCUSS] Numeric Vector type: physical representation (outcome of focused call)

[Daniel Weeks <dw...@apache.org>]

I agree with moving forward with the proposal as-is.

I think what we see from other databases with native vector support is that
they treat them as formal vectors, which distinguishes them from just a
fixed-size array.

It also shows that this is a well-supported type and not some niche type
that is not generally useful/accepted.

-Dan

On Wed, Sep 23, 2026 at 11:04 AM Alkis Evlogimenos via dev <
dev@parquet.apache.org> wrote:

> Engines that disallow nulls, nans and infs:
> - postgress
> - oracle
> - mysql
> - sql server
> - also: milvus, qdrant, pinecone
>
> Notable dbs that allow nulls, nans and infs:
> - duckdb
> - clickhouse
> These seem to have this behavior because they do not have a specific vector
> type so they use array(float) or fixed-size-array(float) and disable
> functionality when nulls or non-finite values are present.
>
>
> So VECTOR across most database engines disallow nulls, nans and infs. This
> means better interoperability between parquet and such dbs so I think we
> should move ahead with the proposal as is.
>
>
> On Wed, Sep 23, 2026 at 5:00 PM Antoine Pitrou <an...@python.org> wrote:
>
> >
> > Le 23/09/2026 à 16:26, Rok Mihevc a écrit :
> > >> Well, Parquet is not a single-purpose format.
> > >>
> > >> Vector stores can choose whatever policy fits their usage, but we're
> > >> talking about a general-purpose file format that aims to be broadly
> > >> applicable.
> > >
> > > What is the gate for Parquet to define something as a logical type?
> >
> > There is no "gate". But we should act in the broader community's
> > interest. A limitation that is only motivated by the needs of a single
> > segment of the user base goes against that goal.
> >
> > >> Why not? Why shouldn't it be used for storing, for example, NumPy
> tensor
> > >> data (which can contain NaNs and infinites)?
> > >
> > > A multidimensional arrays logical type might come with other
> > > requirements (shapes, strides, dimension names), that vector
> > > applications might not have.
> >
> > Well, first, you can use NumPy just for 1D data, and I'm sure some
> > people do that. Second, strides wouldn't be saved in Parquet where they
> > are meaningless. Third, all of this can be conveyed as additional
> > key-value metadata.
> >
> > >> It does not make sense to have *both* a "specialist" Vector type and a
> > >> "general-purpose" FixedSizeList type. Parquet types are a
> > >> general-purpose vocabulary from which you can build up more
> specialized
> > >> applications. Specialist types can be left to domain-specific SW users
> > >> of Parquet.
> > >>
> > >> (for example, Arrow implementations serialize the Arrow schema in
> > >> Parquet metadata so that the Arrow schema can be rebuilt at read time,
> > >> without mandating that every Arrow datatype has a Parquet equivalent)
> > >
> > > That does seem to work nicely for Arrow [0]. Should we revisit the
> > Parquet
> > > extensions discussion [1][2]?
> >
> > I think we should, and then specialized communities such as ML can
> > define their own specialized extension types (with arbitrary semantics)
> > on top of regular Parquet logical/physical types.
> >
> > Regards
> >
> > Antoine.
> >
> >
> >
>

Re: [DISCUSS] Numeric Vector type: physical representation (outcome of focused call)

[Alkis Evlogimenos via dev <de...@parquet.apache.org>]

Engines that disallow nulls, nans and infs:
- postgress
- oracle
- mysql
- sql server
- also: milvus, qdrant, pinecone

Notable dbs that allow nulls, nans and infs:
- duckdb
- clickhouse
These seem to have this behavior because they do not have a specific vector
type so they use array(float) or fixed-size-array(float) and disable
functionality when nulls or non-finite values are present.


So VECTOR across most database engines disallow nulls, nans and infs. This
means better interoperability between parquet and such dbs so I think we
should move ahead with the proposal as is.


On Wed, Sep 23, 2026 at 5:00 PM Antoine Pitrou <an...@python.org> wrote:

>
> Le 23/09/2026 à 16:26, Rok Mihevc a écrit :
> >> Well, Parquet is not a single-purpose format.
> >>
> >> Vector stores can choose whatever policy fits their usage, but we're
> >> talking about a general-purpose file format that aims to be broadly
> >> applicable.
> >
> > What is the gate for Parquet to define something as a logical type?
>
> There is no "gate". But we should act in the broader community's
> interest. A limitation that is only motivated by the needs of a single
> segment of the user base goes against that goal.
>
> >> Why not? Why shouldn't it be used for storing, for example, NumPy tensor
> >> data (which can contain NaNs and infinites)?
> >
> > A multidimensional arrays logical type might come with other
> > requirements (shapes, strides, dimension names), that vector
> > applications might not have.
>
> Well, first, you can use NumPy just for 1D data, and I'm sure some
> people do that. Second, strides wouldn't be saved in Parquet where they
> are meaningless. Third, all of this can be conveyed as additional
> key-value metadata.
>
> >> It does not make sense to have *both* a "specialist" Vector type and a
> >> "general-purpose" FixedSizeList type. Parquet types are a
> >> general-purpose vocabulary from which you can build up more specialized
> >> applications. Specialist types can be left to domain-specific SW users
> >> of Parquet.
> >>
> >> (for example, Arrow implementations serialize the Arrow schema in
> >> Parquet metadata so that the Arrow schema can be rebuilt at read time,
> >> without mandating that every Arrow datatype has a Parquet equivalent)
> >
> > That does seem to work nicely for Arrow [0]. Should we revisit the
> Parquet
> > extensions discussion [1][2]?
>
> I think we should, and then specialized communities such as ML can
> define their own specialized extension types (with arbitrary semantics)
> on top of regular Parquet logical/physical types.
>
> Regards
>
> Antoine.
>
>
>

Re: [DISCUSS] Numeric Vector type: physical representation (outcome of focused call)

[Alkis Evlogimenos via dev <de...@parquet.apache.org>]

Engines that disallow nulls, nans and infs:
- postgress
- oracle
- mysql
- sql server
- also: milvus, qdrant, pinecone

Notable dbs that allow nulls, nans and infs:
- duckdb
- clickhouse
These seem to have this behavior because they do not have a specific vector
type so they use array(float) or fixed-size-array(float) and disable
functionality when nulls or non-finite values are present.


So VECTOR across most database engines disallow nulls, nans and infs. This
means better interoperability between parquet and such dbs so I think we
should move ahead with the proposal as is.


On Wed, Sep 23, 2026 at 5:00 PM Antoine Pitrou <an...@python.org> wrote:

>
> Le 23/09/2026 à 16:26, Rok Mihevc a écrit :
> >> Well, Parquet is not a single-purpose format.
> >>
> >> Vector stores can choose whatever policy fits their usage, but we're
> >> talking about a general-purpose file format that aims to be broadly
> >> applicable.
> >
> > What is the gate for Parquet to define something as a logical type?
>
> There is no "gate". But we should act in the broader community's
> interest. A limitation that is only motivated by the needs of a single
> segment of the user base goes against that goal.
>
> >> Why not? Why shouldn't it be used for storing, for example, NumPy tensor
> >> data (which can contain NaNs and infinites)?
> >
> > A multidimensional arrays logical type might come with other
> > requirements (shapes, strides, dimension names), that vector
> > applications might not have.
>
> Well, first, you can use NumPy just for 1D data, and I'm sure some
> people do that. Second, strides wouldn't be saved in Parquet where they
> are meaningless. Third, all of this can be conveyed as additional
> key-value metadata.
>
> >> It does not make sense to have *both* a "specialist" Vector type and a
> >> "general-purpose" FixedSizeList type. Parquet types are a
> >> general-purpose vocabulary from which you can build up more specialized
> >> applications. Specialist types can be left to domain-specific SW users
> >> of Parquet.
> >>
> >> (for example, Arrow implementations serialize the Arrow schema in
> >> Parquet metadata so that the Arrow schema can be rebuilt at read time,
> >> without mandating that every Arrow datatype has a Parquet equivalent)
> >
> > That does seem to work nicely for Arrow [0]. Should we revisit the
> Parquet
> > extensions discussion [1][2]?
>
> I think we should, and then specialized communities such as ML can
> define their own specialized extension types (with arbitrary semantics)
> on top of regular Parquet logical/physical types.
>
> Regards
>
> Antoine.
>
>
>

Re: [DISCUSS] Numeric Vector type: physical representation (outcome of focused call)

[Antoine Pitrou <an...@python.org>]

Le 23/09/2026 à 16:26, Rok Mihevc a écrit :
>> Well, Parquet is not a single-purpose format.
>>
>> Vector stores can choose whatever policy fits their usage, but we're
>> talking about a general-purpose file format that aims to be broadly
>> applicable.
> 
> What is the gate for Parquet to define something as a logical type?

There is no "gate". But we should act in the broader community's 
interest. A limitation that is only motivated by the needs of a single 
segment of the user base goes against that goal.

>> Why not? Why shouldn't it be used for storing, for example, NumPy tensor
>> data (which can contain NaNs and infinites)?
> 
> A multidimensional arrays logical type might come with other
> requirements (shapes, strides, dimension names), that vector
> applications might not have.

Well, first, you can use NumPy just for 1D data, and I'm sure some 
people do that. Second, strides wouldn't be saved in Parquet where they 
are meaningless. Third, all of this can be conveyed as additional 
key-value metadata.

>> It does not make sense to have *both* a "specialist" Vector type and a
>> "general-purpose" FixedSizeList type. Parquet types are a
>> general-purpose vocabulary from which you can build up more specialized
>> applications. Specialist types can be left to domain-specific SW users
>> of Parquet.
>>
>> (for example, Arrow implementations serialize the Arrow schema in
>> Parquet metadata so that the Arrow schema can be rebuilt at read time,
>> without mandating that every Arrow datatype has a Parquet equivalent)
> 
> That does seem to work nicely for Arrow [0]. Should we revisit the Parquet
> extensions discussion [1][2]?

I think we should, and then specialized communities such as ML can 
define their own specialized extension types (with arbitrary semantics) 
on top of regular Parquet logical/physical types.

Regards

Antoine.

Re: [DISCUSS] Numeric Vector type: physical representation (outcome of focused call)

[Rok Mihevc <ro...@gmail.com>]

> Well, Parquet is not a single-purpose format.
>
> Vector stores can choose whatever policy fits their usage, but we're
> talking about a general-purpose file format that aims to be broadly
> applicable.

What is the gate for Parquet to define something as a logical type?

Regarding the current proposal, would it be acceptable if the finiteness
requirement was removed? Question for everyone!

> Why not? Why shouldn't it be used for storing, for example, NumPy tensor
> data (which can contain NaNs and infinites)?

A multidimensional arrays logical type might come with other
requirements (shapes, strides, dimension names), that vector
applications might not have. We might have to make the common
general type complex or not rich enough to express everything we
need.

> It does not make sense to have *both* a "specialist" Vector type and a
> "general-purpose" FixedSizeList type. Parquet types are a
> general-purpose vocabulary from which you can build up more specialized
> applications. Specialist types can be left to domain-specific SW users
> of Parquet.
>
> (for example, Arrow implementations serialize the Arrow schema in
> Parquet metadata so that the Arrow schema can be rebuilt at read time,
> without mandating that every Arrow datatype has a Parquet equivalent)

That does seem to work nicely for Arrow [0]. Should we revisit the Parquet
extensions discussion [1][2]?

[0]
https://arrow.apache.org/docs/format/CanonicalExtensions.html#official-list
[1] https://lists.apache.org/thread/9xo3mp4n23p8psmrhso5t9q899vxwfjt
[2] https://github.com/apache/parquet-format/pull/451


Best,
Rok

Re: [DISCUSS] Numeric Vector type: physical representation (outcome of focused call)

[Rok Mihevc <ro...@gmail.com>]

> Well, Parquet is not a single-purpose format.
>
> Vector stores can choose whatever policy fits their usage, but we're
> talking about a general-purpose file format that aims to be broadly
> applicable.

What is the gate for Parquet to define something as a logical type?

Regarding the current proposal, would it be acceptable if the finiteness
requirement was removed? Question for everyone!

> Why not? Why shouldn't it be used for storing, for example, NumPy tensor
> data (which can contain NaNs and infinites)?

A multidimensional arrays logical type might come with other
requirements (shapes, strides, dimension names), that vector
applications might not have. We might have to make the common
general type complex or not rich enough to express everything we
need.

> It does not make sense to have *both* a "specialist" Vector type and a
> "general-purpose" FixedSizeList type. Parquet types are a
> general-purpose vocabulary from which you can build up more specialized
> applications. Specialist types can be left to domain-specific SW users
> of Parquet.
>
> (for example, Arrow implementations serialize the Arrow schema in
> Parquet metadata so that the Arrow schema can be rebuilt at read time,
> without mandating that every Arrow datatype has a Parquet equivalent)

That does seem to work nicely for Arrow [0]. Should we revisit the Parquet
extensions discussion [1][2]?

[0]
https://arrow.apache.org/docs/format/CanonicalExtensions.html#official-list
[1] https://lists.apache.org/thread/9xo3mp4n23p8psmrhso5t9q899vxwfjt
[2] https://github.com/apache/parquet-format/pull/451


Best,
Rok

Re: [DISCUSS] Numeric Vector type: physical representation (outcome of focused call)

[Antoine Pitrou <an...@python.org>]

Le 23/09/2026 à 12:09, Rok Mihevc a écrit :
>> The two last items are really gratuitous limitations. Readers are free
>> to implement limitations of their own and refuse to read more general
>> fixed-size-list data, but it does not make sense to impose those
>> limitations in the spec.
> 
> The reason for proposing limitations is that this proposal aims for a
> narrow vector use case (vector databases, embeddings).
> In that context
> non-finite or nested values are invalid. Vector stores typically reject such
> values.

Well, Parquet is not a single-purpose format.

Vector stores can choose whatever policy fits their usage, but we're 
talking about a general-purpose file format that aims to be broadly 
applicable.

> I don't expect applications that need multidimensional array data to use
> this vector logical type

Why not? Why shouldn't it be used for storing, for example, NumPy tensor 
data (which can contain NaNs and infinites)?

> For general fixed-size-list data, I expect we will continue the discussion
> on new physical layout possibilities and provide a general solution there.

Ok, so let's please focus on the general fixed-size-list case then?

It does not make sense to have *both* a "specialist" Vector type and a 
"general-purpose" FixedSizeList type. Parquet types are a 
general-purpose vocabulary from which you can build up more specialized 
applications. Specialist types can be left to domain-specific SW users 
of Parquet.

(for example, Arrow implementations serialize the Arrow schema in 
Parquet metadata so that the Arrow schema can be rebuilt at read time, 
without mandating that every Arrow datatype has a Parquet equivalent)

> I don't think it's too unusual for a narrow logical type to restrict its
> values.
> For instance, JSON requires valid JSON, GEOGRAPHY bounds longitudes
> and latitudes.

"JSON requires valid JSON" is tautological, so I don't think this is a 
valid analogy.

> If we are concerned that writers would have to validate their
> data, we could add an explicit property `optional bool VectorType.finite`.

Why would this be be a property of the Vector type, rather than either 
1) a piece of statistics or 2) a property of the underlying value type?

Regards

Antoine.

Re: [DISCUSS] Numeric Vector type: physical representation (outcome of focused call)

[Rok Mihevc <ro...@gmail.com>]

> The two last items are really gratuitous limitations. Readers are free
> to implement limitations of their own and refuse to read more general
> fixed-size-list data, but it does not make sense to impose those
> limitations in the spec.

The reason for proposing limitations is that this proposal aims for a
narrow vector use case (vector databases, embeddings). In that context
non-finite or nested values are invalid. Vector stores typically reject such
values.

I don't expect applications that need multidimensional array data to use
this vector logical type and I don't think it materially changes the spec
outside of the new vector logical type.

For general fixed-size-list data, I expect we will continue the discussion
on new physical layout possibilities and provide a general solution there.

> (especially the "non-null and finite" item really belongs to statistics
> IMHO)

Non-null is already communicated by the schema.
Regarding finiteness, statistics are sometimes ignored (not written or read)
and require reading metadata, which would require consumers of such type
to be more defensive than necessary.
(finiteness here means not-nan, not-null, not-+/-inf)

I don't think it's too unusual for a narrow logical type to restrict its
values.
For instance, JSON requires valid JSON, GEOGRAPHY bounds longitudes
and latitudes. If we are concerned that writers would have to validate their
data, we could add an explicit property `optional bool VectorType.finite`.


Thank you for reviewing Antoine!

Best,
Rok

Re: [DISCUSS] Numeric Vector type: physical representation (outcome of focused call)

[Rok Mihevc <ro...@gmail.com>]

> The two last items are really gratuitous limitations. Readers are free
> to implement limitations of their own and refuse to read more general
> fixed-size-list data, but it does not make sense to impose those
> limitations in the spec.

The reason for proposing limitations is that this proposal aims for a
narrow vector use case (vector databases, embeddings). In that context
non-finite or nested values are invalid. Vector stores typically reject such
values.

I don't expect applications that need multidimensional array data to use
this vector logical type and I don't think it materially changes the spec
outside of the new vector logical type.

For general fixed-size-list data, I expect we will continue the discussion
on new physical layout possibilities and provide a general solution there.

> (especially the "non-null and finite" item really belongs to statistics
> IMHO)

Non-null is already communicated by the schema.
Regarding finiteness, statistics are sometimes ignored (not written or read)
and require reading metadata, which would require consumers of such type
to be more defensive than necessary.
(finiteness here means not-nan, not-null, not-+/-inf)

I don't think it's too unusual for a narrow logical type to restrict its
values.
For instance, JSON requires valid JSON, GEOGRAPHY bounds longitudes
and latitudes. If we are concerned that writers would have to validate their
data, we could add an explicit property `optional bool VectorType.finite`.


Thank you for reviewing Antoine!

Best,
Rok

Re: [DISCUSS] Numeric Vector type: physical representation (outcome of focused call)

[Antoine Pitrou <an...@python.org>]

Le 22/09/2026 à 18:02, Rok Mihevc a écrit :
> Hi all,
> 
> Following the discussion here and on the community call, I've opened a PR
> for the LIST-based VECTOR logical type:
> 
> https://github.com/apache/parquet-format/pull/624
> 
> This proposal separates the logical contract from the physical-layout
> changes:
> 
> - Every non-null vector has exactly num_elements.
> - Elements must be non-null and finite.
> - Elements are numeric or boolean Parquet primitive types. No nesetd
> elements.

The two last items are really gratuitous limitations. Readers are free 
to implement limitations of their own and refuse to read more general 
fixed-size-list data, but it does not make sense to impose those 
limitations in the spec.

(especially the "non-null and finite" item really belongs to statistics 
IMHO)

Regards

Antoine.

Re: [DISCUSS] Numeric Vector type: physical representation (outcome of focused call)

[Rok Mihevc <ro...@gmail.com>]

Hi all,

Following the discussion here and on the community call, I've opened a PR
for the LIST-based VECTOR logical type:

https://github.com/apache/parquet-format/pull/624

This proposal separates the logical contract from the physical-layout
changes:

- Every non-null vector has exactly num_elements.
- Elements must be non-null and finite.
- Elements are numeric or boolean Parquet primitive types. No nesetd
elements.

Writers must enforce these constraints and readers may rely on them.
VECTOR-unaware readers read VECTOR as ordinary LIST.
Physical layout changes, vector indexes, quantisation and normalisation
guarantees are out of scope for this proposal.


Can we agree to move this logical annotation forward independently of the
physical-layout proposal? Feedback on the scope and details is welcome,
here or on the PR.

This proposal closely resembles Option C in the original design doc [1],
which also contains the earlier discussion.

[1]
https://docs.google.com/document/d/1nf30OqK_UqxA4YTEZQszmOBEG56m9M5mp9rIYC2SUWc/edit?tab=t.0

Best,
Rok

On Wed, Sep 9, 2026 at 1:08 PM Rok Mihevc <ro...@gmail.com> wrote:

> Antoine:
> > > * Vectors (as in vector DB) storage needs are different to what we
> need for
> > > fixed size list applications we've been disucssing so far [0].
> >
> > This seems unsubstantiated. Can you please provide an explanation?
>
> Fair point, "different storage needs" is too broad. What we need from a
> vector logical type is a set of constraints (contract) that we currently
> can't express:
> 1. Vector has a fixed number of numeric elements
> 2. Elements cannot be null, are finite or rather make sense as vectors in
> a space
> 3. Potential future vector specific properties (normalization guarantee,
> statistics) or vector specific encodings
>
> These constraints do not necessarily require a different physical
> representation. For example Lance uses FixedSizeList with specialized
> physical encodings, Hudi specifies vector as a single FLBA [2] in parquet
> with vector semantics carried in metadata.
>
> [1] Lance vector -
> https://github.com/lance-format/lance/blob/main/docs/src/format/table/schema.md#fixed-size-list-types
> [2] Hudi vector -
> https://github.com/apache/hudi/blob/master/rfc/rfc-99/vector-appendix.md#vector-schema-constraints
> ---
>
> Will:
> >   - Vector indexes, along the lines of Lance: how does a reader discover
> > that a column or auxiliary structure indexes another column, or is
> derived
> > from it? What lets an engine recognise when it can use that structure?
> >
> >   - Quantisation, normalisation guarantees, and additional numeric
> formats
> > such as BF16/FP8. We already have FLOAT16 and small integer annotations,
> > but a coordinate type alone doesn't describe all of these.  Some
> quantised
> > representations are naturally composite values: packed coordinates plus
> > scales, offsets, or codebooks. Block-scaled FP4 is a concrete example.
> > These could be structs with separate children for codes and scales, and a
> > logical annotation describing how they collectively represent a vector.
> > Shared parameters might live elsewhere. Either way, the relationship
> needs
> > expressing.
>
> Lance stores the index as a separate structure and table metadata maps the
> index to the vector [3], the index carries information about distance
> metric and quantization [4].
> As per Option C1 we would add new types to parquet's type system. Under
> Option A we would have a vector element-type enum that we could amend.
> Expressing more information like codebooks and quantizations seems to
> require another logical type or other rather than tacking on type
> parameters here.
>
> [3] Lance index-to-column binding -
> https://github.com/lance-format/lance/blob/main/docs/src/format/index/index.md#creating-and-updating-index-segments
> [4] Vector index storage -
> https://github.com/lance-format/lance/blob/main/docs/src/format/index/vector/index.md#storage-layout-v3
> ---
>
> Daniel:
> > I feel like the intent behind having a logical type for vector is to
> > differentiate its semantics from a simple fixed size list.  Vectors
> > shouldn't contain null, NaN, inf as elements (in practice they may
> present,
> > but those are typically due to erroneous handling).  I would expect
> vectors
> > to prohibit special/missing values that render them invalid.  Are the
> > statistics intended to detect these erroneous scenarios or is the intent
> to
> > allow these values?
>
> The intent is to prohibit non-finite values and writers should enforce
> this. However using stats would allow readers to act defensively and
> validate data at read time.
>
> On Wed, Sep 9, 2026 at 1:09 AM Daniel Weeks <dw...@apache.org> wrote:
>
>> Thanks for the summary Rok,
>>
>> I like the direction C1 is going, but I'm a little concerned about the
>> following statement:
>>
>> Regarding NaN/Inf limitations, the consensus was that these should be
>> > handled by statistics. This leads to a need for a new inf_count
>> statustics.
>>
>>
>> I feel like the intent behind having a logical type for vector is to
>> differentiate its semantics from a simple fixed size list.  Vectors
>> shouldn't contain null, NaN, inf as elements (in practice they may
>> present,
>> but those are typically due to erroneous handling).  I would expect
>> vectors
>> to prohibit special/missing values that render them invalid.  Are the
>> statistics intended to detect these erroneous scenarios or is the intent
>> to
>> allow these values?
>>
>> -Dan
>>
>>
>> On Tue, Sep 8, 2026 at 1:41 PM Will Edwards via dev <
>> dev@parquet.apache.org>
>> wrote:
>>
>> > Hi all,
>> >
>> > Andrew's and Gunnar's results show that existing LIST storage can be
>> read
>> > efficiently. This matches my own experience too. I think efficiency is a
>> > software problem not a Parquet format problem.  E.g. the reader can
>> expose
>> > flat typed memory; nothing in the format requires allocating a
>> List<Float>
>> > per row. A nullable schema doesn't change that for pages without nulls,
>> > which the reader can establish at page scope.
>> >
>> >  So is the immediate question what SHOW CREATE TABLE should say when the
>> > only source of schema is a Parquet file? ARRAY<FLOAT>, a fixed-length
>> > array, or VECTOR<FLOAT, 768>? That's a useful discussion about the
>> contract
>> > we're expressing.
>> >
>> > SELECT COS_SIM(a, b) etc can already work on numeric lists, with lengths
>> > checked efficiently through the encoded levels. Whether a and b belong
>> to
>> > the same embedding space remains the user's responsibility; declaring
>> both
>> > to have 768 dimensions doesn't establish that.
>> >
>> > There are other semantics I'd be interested in discussing:
>> >
>> >   - Vector indexes, along the lines of Lance: how does a reader discover
>> > that a column or auxiliary structure indexes another column, or is
>> derived
>> > from it? What lets an engine recognise when it can use that structure?
>> >
>> >   - Quantisation, normalisation guarantees, and additional numeric
>> formats
>> > such as BF16/FP8. We already have FLOAT16 and small integer annotations,
>> > but a coordinate type alone doesn't describe all of these.  Some
>> quantised
>> > representations are naturally composite values: packed coordinates plus
>> > scales, offsets, or codebooks. Block-scaled FP4 is a concrete example.
>> > These could be structs with separate children for codes and scales, and
>> a
>> > logical annotation describing how they collectively represent a vector.
>> > Shared parameters might live elsewhere. Either way, the relationship
>> needs
>> > expressing.
>> >
>> > Should we pin down those requirements before choosing a physical
>> > representation?  Adding an element-type enum doesn't answer all of
>> these,
>> > and packing whole vectors into FLBA limits existing element encoding
>> > choices.
>> >
>> >  What concrete vector workload still needs a physical change after
>> applying
>> > the reader optimisations we've discussed?
>> >
>> >  Will
>> >
>> > On Tue, 8 Sept 2026 at 21:22, Antoine Pitrou <an...@python.org>
>> wrote:
>> >
>> > >
>> > > Le 08/09/2026 à 20:42, Rok Mihevc a écrit :
>> > > > Hi all,
>> > > >
>> > > > As proposed during the general community call we held a focused call
>> > > today
>> > > > to discuss the proposal for adding a logical vector type to parquet.
>> > This
>> > > > was called because it seems:
>> > > >
>> > > > * Vectors (as in vector DB) storage needs are different to what we
>> need
>> > > for
>> > > > fixed size list applications we've been disucssing so far [0].
>> > >
>> > > This seems unsubstantiated. Can you please provide an explanation?
>> > >
>> > > Ideally we'd use the *same* underlying solution for both problems.
>> > >
>> > > > We have converged on two options that could work, but have not yet
>> > > decided
>> > > > which one to propose:
>> > > >
>> > > > - Option A - FLBA as physical storage (FLBA(num_elements *
>> > > element_width)).
>> > > > Vector logical type would provide an enum specifying the element
>> type,
>> > > and
>> > > > the element would be non-nullable. The element type enum could be
>> > > > independent of the parquet type system, making evolution easier.
>> > Downside
>> > > > is loss of element-level encodings available today, though they can
>> be
>> > > > added later. Upside is simplicity and ease of evolution.
>> > > >
>> > > > - Option C1 - LIST with numeric parquet type elements as physical
>> > > storage.
>> > > > New float types (e.g. BF16) can be added in parquet proper as
>> needed.
>> > > > Downside is we need to add new types to parquet, DL/RL overhead is
>> > likely
>> > > > not a concern. Upside is we have encoding support now.
>> > >
>> > > Option A is a short-term solution with severe limitations (lack of
>> > > encoding choices) that make me entirely negative about the idea.
>> > >
>> > > I'm lukewarm about Option C (especially the optimistic statement that
>> > > "DL/RL overhead is likely not a concern"), but at least it doesn't
>> paint
>> > > us in a corner.
>> > >
>> > > Regards
>> > >
>> > > Antoine.
>> > >
>> > >
>> > >
>> >
>>
>

Re: [DISCUSS] Numeric Vector type: physical representation (outcome of focused call)

[Rok Mihevc <ro...@gmail.com>]

Hi all,

Following the discussion here and on the community call, I've opened a PR
for the LIST-based VECTOR logical type:

https://github.com/apache/parquet-format/pull/624

This proposal separates the logical contract from the physical-layout
changes:

- Every non-null vector has exactly num_elements.
- Elements must be non-null and finite.
- Elements are numeric or boolean Parquet primitive types. No nesetd
elements.

Writers must enforce these constraints and readers may rely on them.
VECTOR-unaware readers read VECTOR as ordinary LIST.
Physical layout changes, vector indexes, quantisation and normalisation
guarantees are out of scope for this proposal.


Can we agree to move this logical annotation forward independently of the
physical-layout proposal? Feedback on the scope and details is welcome,
here or on the PR.

This proposal closely resembles Option C in the original design doc [1],
which also contains the earlier discussion.

[1]
https://docs.google.com/document/d/1nf30OqK_UqxA4YTEZQszmOBEG56m9M5mp9rIYC2SUWc/edit?tab=t.0

Best,
Rok

On Wed, Sep 9, 2026 at 1:08 PM Rok Mihevc <ro...@gmail.com> wrote:

> Antoine:
> > > * Vectors (as in vector DB) storage needs are different to what we
> need for
> > > fixed size list applications we've been disucssing so far [0].
> >
> > This seems unsubstantiated. Can you please provide an explanation?
>
> Fair point, "different storage needs" is too broad. What we need from a
> vector logical type is a set of constraints (contract) that we currently
> can't express:
> 1. Vector has a fixed number of numeric elements
> 2. Elements cannot be null, are finite or rather make sense as vectors in
> a space
> 3. Potential future vector specific properties (normalization guarantee,
> statistics) or vector specific encodings
>
> These constraints do not necessarily require a different physical
> representation. For example Lance uses FixedSizeList with specialized
> physical encodings, Hudi specifies vector as a single FLBA [2] in parquet
> with vector semantics carried in metadata.
>
> [1] Lance vector -
> https://github.com/lance-format/lance/blob/main/docs/src/format/table/schema.md#fixed-size-list-types
> [2] Hudi vector -
> https://github.com/apache/hudi/blob/master/rfc/rfc-99/vector-appendix.md#vector-schema-constraints
> ---
>
> Will:
> >   - Vector indexes, along the lines of Lance: how does a reader discover
> > that a column or auxiliary structure indexes another column, or is
> derived
> > from it? What lets an engine recognise when it can use that structure?
> >
> >   - Quantisation, normalisation guarantees, and additional numeric
> formats
> > such as BF16/FP8. We already have FLOAT16 and small integer annotations,
> > but a coordinate type alone doesn't describe all of these.  Some
> quantised
> > representations are naturally composite values: packed coordinates plus
> > scales, offsets, or codebooks. Block-scaled FP4 is a concrete example.
> > These could be structs with separate children for codes and scales, and a
> > logical annotation describing how they collectively represent a vector.
> > Shared parameters might live elsewhere. Either way, the relationship
> needs
> > expressing.
>
> Lance stores the index as a separate structure and table metadata maps the
> index to the vector [3], the index carries information about distance
> metric and quantization [4].
> As per Option C1 we would add new types to parquet's type system. Under
> Option A we would have a vector element-type enum that we could amend.
> Expressing more information like codebooks and quantizations seems to
> require another logical type or other rather than tacking on type
> parameters here.
>
> [3] Lance index-to-column binding -
> https://github.com/lance-format/lance/blob/main/docs/src/format/index/index.md#creating-and-updating-index-segments
> [4] Vector index storage -
> https://github.com/lance-format/lance/blob/main/docs/src/format/index/vector/index.md#storage-layout-v3
> ---
>
> Daniel:
> > I feel like the intent behind having a logical type for vector is to
> > differentiate its semantics from a simple fixed size list.  Vectors
> > shouldn't contain null, NaN, inf as elements (in practice they may
> present,
> > but those are typically due to erroneous handling).  I would expect
> vectors
> > to prohibit special/missing values that render them invalid.  Are the
> > statistics intended to detect these erroneous scenarios or is the intent
> to
> > allow these values?
>
> The intent is to prohibit non-finite values and writers should enforce
> this. However using stats would allow readers to act defensively and
> validate data at read time.
>
> On Wed, Sep 9, 2026 at 1:09 AM Daniel Weeks <dw...@apache.org> wrote:
>
>> Thanks for the summary Rok,
>>
>> I like the direction C1 is going, but I'm a little concerned about the
>> following statement:
>>
>> Regarding NaN/Inf limitations, the consensus was that these should be
>> > handled by statistics. This leads to a need for a new inf_count
>> statustics.
>>
>>
>> I feel like the intent behind having a logical type for vector is to
>> differentiate its semantics from a simple fixed size list.  Vectors
>> shouldn't contain null, NaN, inf as elements (in practice they may
>> present,
>> but those are typically due to erroneous handling).  I would expect
>> vectors
>> to prohibit special/missing values that render them invalid.  Are the
>> statistics intended to detect these erroneous scenarios or is the intent
>> to
>> allow these values?
>>
>> -Dan
>>
>>
>> On Tue, Sep 8, 2026 at 1:41 PM Will Edwards via dev <
>> dev@parquet.apache.org>
>> wrote:
>>
>> > Hi all,
>> >
>> > Andrew's and Gunnar's results show that existing LIST storage can be
>> read
>> > efficiently. This matches my own experience too. I think efficiency is a
>> > software problem not a Parquet format problem.  E.g. the reader can
>> expose
>> > flat typed memory; nothing in the format requires allocating a
>> List<Float>
>> > per row. A nullable schema doesn't change that for pages without nulls,
>> > which the reader can establish at page scope.
>> >
>> >  So is the immediate question what SHOW CREATE TABLE should say when the
>> > only source of schema is a Parquet file? ARRAY<FLOAT>, a fixed-length
>> > array, or VECTOR<FLOAT, 768>? That's a useful discussion about the
>> contract
>> > we're expressing.
>> >
>> > SELECT COS_SIM(a, b) etc can already work on numeric lists, with lengths
>> > checked efficiently through the encoded levels. Whether a and b belong
>> to
>> > the same embedding space remains the user's responsibility; declaring
>> both
>> > to have 768 dimensions doesn't establish that.
>> >
>> > There are other semantics I'd be interested in discussing:
>> >
>> >   - Vector indexes, along the lines of Lance: how does a reader discover
>> > that a column or auxiliary structure indexes another column, or is
>> derived
>> > from it? What lets an engine recognise when it can use that structure?
>> >
>> >   - Quantisation, normalisation guarantees, and additional numeric
>> formats
>> > such as BF16/FP8. We already have FLOAT16 and small integer annotations,
>> > but a coordinate type alone doesn't describe all of these.  Some
>> quantised
>> > representations are naturally composite values: packed coordinates plus
>> > scales, offsets, or codebooks. Block-scaled FP4 is a concrete example.
>> > These could be structs with separate children for codes and scales, and
>> a
>> > logical annotation describing how they collectively represent a vector.
>> > Shared parameters might live elsewhere. Either way, the relationship
>> needs
>> > expressing.
>> >
>> > Should we pin down those requirements before choosing a physical
>> > representation?  Adding an element-type enum doesn't answer all of
>> these,
>> > and packing whole vectors into FLBA limits existing element encoding
>> > choices.
>> >
>> >  What concrete vector workload still needs a physical change after
>> applying
>> > the reader optimisations we've discussed?
>> >
>> >  Will
>> >
>> > On Tue, 8 Sept 2026 at 21:22, Antoine Pitrou <an...@python.org>
>> wrote:
>> >
>> > >
>> > > Le 08/09/2026 à 20:42, Rok Mihevc a écrit :
>> > > > Hi all,
>> > > >
>> > > > As proposed during the general community call we held a focused call
>> > > today
>> > > > to discuss the proposal for adding a logical vector type to parquet.
>> > This
>> > > > was called because it seems:
>> > > >
>> > > > * Vectors (as in vector DB) storage needs are different to what we
>> need
>> > > for
>> > > > fixed size list applications we've been disucssing so far [0].
>> > >
>> > > This seems unsubstantiated. Can you please provide an explanation?
>> > >
>> > > Ideally we'd use the *same* underlying solution for both problems.
>> > >
>> > > > We have converged on two options that could work, but have not yet
>> > > decided
>> > > > which one to propose:
>> > > >
>> > > > - Option A - FLBA as physical storage (FLBA(num_elements *
>> > > element_width)).
>> > > > Vector logical type would provide an enum specifying the element
>> type,
>> > > and
>> > > > the element would be non-nullable. The element type enum could be
>> > > > independent of the parquet type system, making evolution easier.
>> > Downside
>> > > > is loss of element-level encodings available today, though they can
>> be
>> > > > added later. Upside is simplicity and ease of evolution.
>> > > >
>> > > > - Option C1 - LIST with numeric parquet type elements as physical
>> > > storage.
>> > > > New float types (e.g. BF16) can be added in parquet proper as
>> needed.
>> > > > Downside is we need to add new types to parquet, DL/RL overhead is
>> > likely
>> > > > not a concern. Upside is we have encoding support now.
>> > >
>> > > Option A is a short-term solution with severe limitations (lack of
>> > > encoding choices) that make me entirely negative about the idea.
>> > >
>> > > I'm lukewarm about Option C (especially the optimistic statement that
>> > > "DL/RL overhead is likely not a concern"), but at least it doesn't
>> paint
>> > > us in a corner.
>> > >
>> > > Regards
>> > >
>> > > Antoine.
>> > >
>> > >
>> > >
>> >
>>
>

Re: [DISCUSS] Numeric Vector type: physical representation (outcome of focused call)

[Rok Mihevc <ro...@gmail.com>]

Antoine:
> > * Vectors (as in vector DB) storage needs are different to what we need
for
> > fixed size list applications we've been disucssing so far [0].
>
> This seems unsubstantiated. Can you please provide an explanation?

Fair point, "different storage needs" is too broad. What we need from a
vector logical type is a set of constraints (contract) that we currently
can't express:
1. Vector has a fixed number of numeric elements
2. Elements cannot be null, are finite or rather make sense as vectors in a
space
3. Potential future vector specific properties (normalization guarantee,
statistics) or vector specific encodings

These constraints do not necessarily require a different physical
representation. For example Lance uses FixedSizeList with specialized
physical encodings, Hudi specifies vector as a single FLBA [2] in parquet
with vector semantics carried in metadata.

[1] Lance vector -
https://github.com/lance-format/lance/blob/main/docs/src/format/table/schema.md#fixed-size-list-types
[2] Hudi vector -
https://github.com/apache/hudi/blob/master/rfc/rfc-99/vector-appendix.md#vector-schema-constraints
---

Will:
>   - Vector indexes, along the lines of Lance: how does a reader discover
> that a column or auxiliary structure indexes another column, or is derived
> from it? What lets an engine recognise when it can use that structure?
>
>   - Quantisation, normalisation guarantees, and additional numeric formats
> such as BF16/FP8. We already have FLOAT16 and small integer annotations,
> but a coordinate type alone doesn't describe all of these.  Some quantised
> representations are naturally composite values: packed coordinates plus
> scales, offsets, or codebooks. Block-scaled FP4 is a concrete example.
> These could be structs with separate children for codes and scales, and a
> logical annotation describing how they collectively represent a vector.
> Shared parameters might live elsewhere. Either way, the relationship needs
> expressing.

Lance stores the index as a separate structure and table metadata maps the
index to the vector [3], the index carries information about distance
metric and quantization [4].
As per Option C1 we would add new types to parquet's type system. Under
Option A we would have a vector element-type enum that we could amend.
Expressing more information like codebooks and quantizations seems to
require another logical type or other rather than tacking on type
parameters here.

[3] Lance index-to-column binding -
https://github.com/lance-format/lance/blob/main/docs/src/format/index/index.md#creating-and-updating-index-segments
[4] Vector index storage -
https://github.com/lance-format/lance/blob/main/docs/src/format/index/vector/index.md#storage-layout-v3
---

Daniel:
> I feel like the intent behind having a logical type for vector is to
> differentiate its semantics from a simple fixed size list.  Vectors
> shouldn't contain null, NaN, inf as elements (in practice they may
present,
> but those are typically due to erroneous handling).  I would expect
vectors
> to prohibit special/missing values that render them invalid.  Are the
> statistics intended to detect these erroneous scenarios or is the intent
to
> allow these values?

The intent is to prohibit non-finite values and writers should enforce
this. However using stats would allow readers to act defensively and
validate data at read time.

On Wed, Sep 9, 2026 at 1:09 AM Daniel Weeks <dw...@apache.org> wrote:

> Thanks for the summary Rok,
>
> I like the direction C1 is going, but I'm a little concerned about the
> following statement:
>
> Regarding NaN/Inf limitations, the consensus was that these should be
> > handled by statistics. This leads to a need for a new inf_count
> statustics.
>
>
> I feel like the intent behind having a logical type for vector is to
> differentiate its semantics from a simple fixed size list.  Vectors
> shouldn't contain null, NaN, inf as elements (in practice they may present,
> but those are typically due to erroneous handling).  I would expect vectors
> to prohibit special/missing values that render them invalid.  Are the
> statistics intended to detect these erroneous scenarios or is the intent to
> allow these values?
>
> -Dan
>
>
> On Tue, Sep 8, 2026 at 1:41 PM Will Edwards via dev <
> dev@parquet.apache.org>
> wrote:
>
> > Hi all,
> >
> > Andrew's and Gunnar's results show that existing LIST storage can be read
> > efficiently. This matches my own experience too. I think efficiency is a
> > software problem not a Parquet format problem.  E.g. the reader can
> expose
> > flat typed memory; nothing in the format requires allocating a
> List<Float>
> > per row. A nullable schema doesn't change that for pages without nulls,
> > which the reader can establish at page scope.
> >
> >  So is the immediate question what SHOW CREATE TABLE should say when the
> > only source of schema is a Parquet file? ARRAY<FLOAT>, a fixed-length
> > array, or VECTOR<FLOAT, 768>? That's a useful discussion about the
> contract
> > we're expressing.
> >
> > SELECT COS_SIM(a, b) etc can already work on numeric lists, with lengths
> > checked efficiently through the encoded levels. Whether a and b belong to
> > the same embedding space remains the user's responsibility; declaring
> both
> > to have 768 dimensions doesn't establish that.
> >
> > There are other semantics I'd be interested in discussing:
> >
> >   - Vector indexes, along the lines of Lance: how does a reader discover
> > that a column or auxiliary structure indexes another column, or is
> derived
> > from it? What lets an engine recognise when it can use that structure?
> >
> >   - Quantisation, normalisation guarantees, and additional numeric
> formats
> > such as BF16/FP8. We already have FLOAT16 and small integer annotations,
> > but a coordinate type alone doesn't describe all of these.  Some
> quantised
> > representations are naturally composite values: packed coordinates plus
> > scales, offsets, or codebooks. Block-scaled FP4 is a concrete example.
> > These could be structs with separate children for codes and scales, and a
> > logical annotation describing how they collectively represent a vector.
> > Shared parameters might live elsewhere. Either way, the relationship
> needs
> > expressing.
> >
> > Should we pin down those requirements before choosing a physical
> > representation?  Adding an element-type enum doesn't answer all of these,
> > and packing whole vectors into FLBA limits existing element encoding
> > choices.
> >
> >  What concrete vector workload still needs a physical change after
> applying
> > the reader optimisations we've discussed?
> >
> >  Will
> >
> > On Tue, 8 Sept 2026 at 21:22, Antoine Pitrou <an...@python.org> wrote:
> >
> > >
> > > Le 08/09/2026 à 20:42, Rok Mihevc a écrit :
> > > > Hi all,
> > > >
> > > > As proposed during the general community call we held a focused call
> > > today
> > > > to discuss the proposal for adding a logical vector type to parquet.
> > This
> > > > was called because it seems:
> > > >
> > > > * Vectors (as in vector DB) storage needs are different to what we
> need
> > > for
> > > > fixed size list applications we've been disucssing so far [0].
> > >
> > > This seems unsubstantiated. Can you please provide an explanation?
> > >
> > > Ideally we'd use the *same* underlying solution for both problems.
> > >
> > > > We have converged on two options that could work, but have not yet
> > > decided
> > > > which one to propose:
> > > >
> > > > - Option A - FLBA as physical storage (FLBA(num_elements *
> > > element_width)).
> > > > Vector logical type would provide an enum specifying the element
> type,
> > > and
> > > > the element would be non-nullable. The element type enum could be
> > > > independent of the parquet type system, making evolution easier.
> > Downside
> > > > is loss of element-level encodings available today, though they can
> be
> > > > added later. Upside is simplicity and ease of evolution.
> > > >
> > > > - Option C1 - LIST with numeric parquet type elements as physical
> > > storage.
> > > > New float types (e.g. BF16) can be added in parquet proper as needed.
> > > > Downside is we need to add new types to parquet, DL/RL overhead is
> > likely
> > > > not a concern. Upside is we have encoding support now.
> > >
> > > Option A is a short-term solution with severe limitations (lack of
> > > encoding choices) that make me entirely negative about the idea.
> > >
> > > I'm lukewarm about Option C (especially the optimistic statement that
> > > "DL/RL overhead is likely not a concern"), but at least it doesn't
> paint
> > > us in a corner.
> > >
> > > Regards
> > >
> > > Antoine.
> > >
> > >
> > >
> >
>

Re: [DISCUSS] Numeric Vector type: physical representation (outcome of focused call)

[Rok Mihevc <ro...@gmail.com>]

Antoine:
> > * Vectors (as in vector DB) storage needs are different to what we need
for
> > fixed size list applications we've been disucssing so far [0].
>
> This seems unsubstantiated. Can you please provide an explanation?

Fair point, "different storage needs" is too broad. What we need from a
vector logical type is a set of constraints (contract) that we currently
can't express:
1. Vector has a fixed number of numeric elements
2. Elements cannot be null, are finite or rather make sense as vectors in a
space
3. Potential future vector specific properties (normalization guarantee,
statistics) or vector specific encodings

These constraints do not necessarily require a different physical
representation. For example Lance uses FixedSizeList with specialized
physical encodings, Hudi specifies vector as a single FLBA [2] in parquet
with vector semantics carried in metadata.

[1] Lance vector -
https://github.com/lance-format/lance/blob/main/docs/src/format/table/schema.md#fixed-size-list-types
[2] Hudi vector -
https://github.com/apache/hudi/blob/master/rfc/rfc-99/vector-appendix.md#vector-schema-constraints
---

Will:
>   - Vector indexes, along the lines of Lance: how does a reader discover
> that a column or auxiliary structure indexes another column, or is derived
> from it? What lets an engine recognise when it can use that structure?
>
>   - Quantisation, normalisation guarantees, and additional numeric formats
> such as BF16/FP8. We already have FLOAT16 and small integer annotations,
> but a coordinate type alone doesn't describe all of these.  Some quantised
> representations are naturally composite values: packed coordinates plus
> scales, offsets, or codebooks. Block-scaled FP4 is a concrete example.
> These could be structs with separate children for codes and scales, and a
> logical annotation describing how they collectively represent a vector.
> Shared parameters might live elsewhere. Either way, the relationship needs
> expressing.

Lance stores the index as a separate structure and table metadata maps the
index to the vector [3], the index carries information about distance
metric and quantization [4].
As per Option C1 we would add new types to parquet's type system. Under
Option A we would have a vector element-type enum that we could amend.
Expressing more information like codebooks and quantizations seems to
require another logical type or other rather than tacking on type
parameters here.

[3] Lance index-to-column binding -
https://github.com/lance-format/lance/blob/main/docs/src/format/index/index.md#creating-and-updating-index-segments
[4] Vector index storage -
https://github.com/lance-format/lance/blob/main/docs/src/format/index/vector/index.md#storage-layout-v3
---

Daniel:
> I feel like the intent behind having a logical type for vector is to
> differentiate its semantics from a simple fixed size list.  Vectors
> shouldn't contain null, NaN, inf as elements (in practice they may
present,
> but those are typically due to erroneous handling).  I would expect
vectors
> to prohibit special/missing values that render them invalid.  Are the
> statistics intended to detect these erroneous scenarios or is the intent
to
> allow these values?

The intent is to prohibit non-finite values and writers should enforce
this. However using stats would allow readers to act defensively and
validate data at read time.

On Wed, Sep 9, 2026 at 1:09 AM Daniel Weeks <dw...@apache.org> wrote:

> Thanks for the summary Rok,
>
> I like the direction C1 is going, but I'm a little concerned about the
> following statement:
>
> Regarding NaN/Inf limitations, the consensus was that these should be
> > handled by statistics. This leads to a need for a new inf_count
> statustics.
>
>
> I feel like the intent behind having a logical type for vector is to
> differentiate its semantics from a simple fixed size list.  Vectors
> shouldn't contain null, NaN, inf as elements (in practice they may present,
> but those are typically due to erroneous handling).  I would expect vectors
> to prohibit special/missing values that render them invalid.  Are the
> statistics intended to detect these erroneous scenarios or is the intent to
> allow these values?
>
> -Dan
>
>
> On Tue, Sep 8, 2026 at 1:41 PM Will Edwards via dev <
> dev@parquet.apache.org>
> wrote:
>
> > Hi all,
> >
> > Andrew's and Gunnar's results show that existing LIST storage can be read
> > efficiently. This matches my own experience too. I think efficiency is a
> > software problem not a Parquet format problem.  E.g. the reader can
> expose
> > flat typed memory; nothing in the format requires allocating a
> List<Float>
> > per row. A nullable schema doesn't change that for pages without nulls,
> > which the reader can establish at page scope.
> >
> >  So is the immediate question what SHOW CREATE TABLE should say when the
> > only source of schema is a Parquet file? ARRAY<FLOAT>, a fixed-length
> > array, or VECTOR<FLOAT, 768>? That's a useful discussion about the
> contract
> > we're expressing.
> >
> > SELECT COS_SIM(a, b) etc can already work on numeric lists, with lengths
> > checked efficiently through the encoded levels. Whether a and b belong to
> > the same embedding space remains the user's responsibility; declaring
> both
> > to have 768 dimensions doesn't establish that.
> >
> > There are other semantics I'd be interested in discussing:
> >
> >   - Vector indexes, along the lines of Lance: how does a reader discover
> > that a column or auxiliary structure indexes another column, or is
> derived
> > from it? What lets an engine recognise when it can use that structure?
> >
> >   - Quantisation, normalisation guarantees, and additional numeric
> formats
> > such as BF16/FP8. We already have FLOAT16 and small integer annotations,
> > but a coordinate type alone doesn't describe all of these.  Some
> quantised
> > representations are naturally composite values: packed coordinates plus
> > scales, offsets, or codebooks. Block-scaled FP4 is a concrete example.
> > These could be structs with separate children for codes and scales, and a
> > logical annotation describing how they collectively represent a vector.
> > Shared parameters might live elsewhere. Either way, the relationship
> needs
> > expressing.
> >
> > Should we pin down those requirements before choosing a physical
> > representation?  Adding an element-type enum doesn't answer all of these,
> > and packing whole vectors into FLBA limits existing element encoding
> > choices.
> >
> >  What concrete vector workload still needs a physical change after
> applying
> > the reader optimisations we've discussed?
> >
> >  Will
> >
> > On Tue, 8 Sept 2026 at 21:22, Antoine Pitrou <an...@python.org> wrote:
> >
> > >
> > > Le 08/09/2026 à 20:42, Rok Mihevc a écrit :
> > > > Hi all,
> > > >
> > > > As proposed during the general community call we held a focused call
> > > today
> > > > to discuss the proposal for adding a logical vector type to parquet.
> > This
> > > > was called because it seems:
> > > >
> > > > * Vectors (as in vector DB) storage needs are different to what we
> need
> > > for
> > > > fixed size list applications we've been disucssing so far [0].
> > >
> > > This seems unsubstantiated. Can you please provide an explanation?
> > >
> > > Ideally we'd use the *same* underlying solution for both problems.
> > >
> > > > We have converged on two options that could work, but have not yet
> > > decided
> > > > which one to propose:
> > > >
> > > > - Option A - FLBA as physical storage (FLBA(num_elements *
> > > element_width)).
> > > > Vector logical type would provide an enum specifying the element
> type,
> > > and
> > > > the element would be non-nullable. The element type enum could be
> > > > independent of the parquet type system, making evolution easier.
> > Downside
> > > > is loss of element-level encodings available today, though they can
> be
> > > > added later. Upside is simplicity and ease of evolution.
> > > >
> > > > - Option C1 - LIST with numeric parquet type elements as physical
> > > storage.
> > > > New float types (e.g. BF16) can be added in parquet proper as needed.
> > > > Downside is we need to add new types to parquet, DL/RL overhead is
> > likely
> > > > not a concern. Upside is we have encoding support now.
> > >
> > > Option A is a short-term solution with severe limitations (lack of
> > > encoding choices) that make me entirely negative about the idea.
> > >
> > > I'm lukewarm about Option C (especially the optimistic statement that
> > > "DL/RL overhead is likely not a concern"), but at least it doesn't
> paint
> > > us in a corner.
> > >
> > > Regards
> > >
> > > Antoine.
> > >
> > >
> > >
> >
>

Re: [DISCUSS] Numeric Vector type: physical representation (outcome of focused call)

[Daniel Weeks <dw...@apache.org>]

Thanks for the summary Rok,

I like the direction C1 is going, but I'm a little concerned about the
following statement:

Regarding NaN/Inf limitations, the consensus was that these should be
> handled by statistics. This leads to a need for a new inf_count statustics.


I feel like the intent behind having a logical type for vector is to
differentiate its semantics from a simple fixed size list.  Vectors
shouldn't contain null, NaN, inf as elements (in practice they may present,
but those are typically due to erroneous handling).  I would expect vectors
to prohibit special/missing values that render them invalid.  Are the
statistics intended to detect these erroneous scenarios or is the intent to
allow these values?

-Dan


On Tue, Sep 8, 2026 at 1:41 PM Will Edwards via dev <de...@parquet.apache.org>
wrote:

> Hi all,
>
> Andrew's and Gunnar's results show that existing LIST storage can be read
> efficiently. This matches my own experience too. I think efficiency is a
> software problem not a Parquet format problem.  E.g. the reader can expose
> flat typed memory; nothing in the format requires allocating a List<Float>
> per row. A nullable schema doesn't change that for pages without nulls,
> which the reader can establish at page scope.
>
>  So is the immediate question what SHOW CREATE TABLE should say when the
> only source of schema is a Parquet file? ARRAY<FLOAT>, a fixed-length
> array, or VECTOR<FLOAT, 768>? That's a useful discussion about the contract
> we're expressing.
>
> SELECT COS_SIM(a, b) etc can already work on numeric lists, with lengths
> checked efficiently through the encoded levels. Whether a and b belong to
> the same embedding space remains the user's responsibility; declaring both
> to have 768 dimensions doesn't establish that.
>
> There are other semantics I'd be interested in discussing:
>
>   - Vector indexes, along the lines of Lance: how does a reader discover
> that a column or auxiliary structure indexes another column, or is derived
> from it? What lets an engine recognise when it can use that structure?
>
>   - Quantisation, normalisation guarantees, and additional numeric formats
> such as BF16/FP8. We already have FLOAT16 and small integer annotations,
> but a coordinate type alone doesn't describe all of these.  Some quantised
> representations are naturally composite values: packed coordinates plus
> scales, offsets, or codebooks. Block-scaled FP4 is a concrete example.
> These could be structs with separate children for codes and scales, and a
> logical annotation describing how they collectively represent a vector.
> Shared parameters might live elsewhere. Either way, the relationship needs
> expressing.
>
> Should we pin down those requirements before choosing a physical
> representation?  Adding an element-type enum doesn't answer all of these,
> and packing whole vectors into FLBA limits existing element encoding
> choices.
>
>  What concrete vector workload still needs a physical change after applying
> the reader optimisations we've discussed?
>
>  Will
>
> On Tue, 8 Sept 2026 at 21:22, Antoine Pitrou <an...@python.org> wrote:
>
> >
> > Le 08/09/2026 à 20:42, Rok Mihevc a écrit :
> > > Hi all,
> > >
> > > As proposed during the general community call we held a focused call
> > today
> > > to discuss the proposal for adding a logical vector type to parquet.
> This
> > > was called because it seems:
> > >
> > > * Vectors (as in vector DB) storage needs are different to what we need
> > for
> > > fixed size list applications we've been disucssing so far [0].
> >
> > This seems unsubstantiated. Can you please provide an explanation?
> >
> > Ideally we'd use the *same* underlying solution for both problems.
> >
> > > We have converged on two options that could work, but have not yet
> > decided
> > > which one to propose:
> > >
> > > - Option A - FLBA as physical storage (FLBA(num_elements *
> > element_width)).
> > > Vector logical type would provide an enum specifying the element type,
> > and
> > > the element would be non-nullable. The element type enum could be
> > > independent of the parquet type system, making evolution easier.
> Downside
> > > is loss of element-level encodings available today, though they can be
> > > added later. Upside is simplicity and ease of evolution.
> > >
> > > - Option C1 - LIST with numeric parquet type elements as physical
> > storage.
> > > New float types (e.g. BF16) can be added in parquet proper as needed.
> > > Downside is we need to add new types to parquet, DL/RL overhead is
> likely
> > > not a concern. Upside is we have encoding support now.
> >
> > Option A is a short-term solution with severe limitations (lack of
> > encoding choices) that make me entirely negative about the idea.
> >
> > I'm lukewarm about Option C (especially the optimistic statement that
> > "DL/RL overhead is likely not a concern"), but at least it doesn't paint
> > us in a corner.
> >
> > Regards
> >
> > Antoine.
> >
> >
> >
>

Re: [DISCUSS] Numeric Vector type: physical representation (outcome of focused call)

[Will Edwards via dev <de...@parquet.apache.org>]

Hi all,

Andrew's and Gunnar's results show that existing LIST storage can be read
efficiently. This matches my own experience too. I think efficiency is a
software problem not a Parquet format problem.  E.g. the reader can expose
flat typed memory; nothing in the format requires allocating a List<Float>
per row. A nullable schema doesn't change that for pages without nulls,
which the reader can establish at page scope.

 So is the immediate question what SHOW CREATE TABLE should say when the
only source of schema is a Parquet file? ARRAY<FLOAT>, a fixed-length
array, or VECTOR<FLOAT, 768>? That's a useful discussion about the contract
we're expressing.

SELECT COS_SIM(a, b) etc can already work on numeric lists, with lengths
checked efficiently through the encoded levels. Whether a and b belong to
the same embedding space remains the user's responsibility; declaring both
to have 768 dimensions doesn't establish that.

There are other semantics I'd be interested in discussing:

  - Vector indexes, along the lines of Lance: how does a reader discover
that a column or auxiliary structure indexes another column, or is derived
from it? What lets an engine recognise when it can use that structure?

  - Quantisation, normalisation guarantees, and additional numeric formats
such as BF16/FP8. We already have FLOAT16 and small integer annotations,
but a coordinate type alone doesn't describe all of these.  Some quantised
representations are naturally composite values: packed coordinates plus
scales, offsets, or codebooks. Block-scaled FP4 is a concrete example.
These could be structs with separate children for codes and scales, and a
logical annotation describing how they collectively represent a vector.
Shared parameters might live elsewhere. Either way, the relationship needs
expressing.

Should we pin down those requirements before choosing a physical
representation?  Adding an element-type enum doesn't answer all of these,
and packing whole vectors into FLBA limits existing element encoding
choices.

 What concrete vector workload still needs a physical change after applying
the reader optimisations we've discussed?

 Will

On Tue, 8 Sept 2026 at 21:22, Antoine Pitrou <an...@python.org> wrote:

>
> Le 08/09/2026 à 20:42, Rok Mihevc a écrit :
> > Hi all,
> >
> > As proposed during the general community call we held a focused call
> today
> > to discuss the proposal for adding a logical vector type to parquet. This
> > was called because it seems:
> >
> > * Vectors (as in vector DB) storage needs are different to what we need
> for
> > fixed size list applications we've been disucssing so far [0].
>
> This seems unsubstantiated. Can you please provide an explanation?
>
> Ideally we'd use the *same* underlying solution for both problems.
>
> > We have converged on two options that could work, but have not yet
> decided
> > which one to propose:
> >
> > - Option A - FLBA as physical storage (FLBA(num_elements *
> element_width)).
> > Vector logical type would provide an enum specifying the element type,
> and
> > the element would be non-nullable. The element type enum could be
> > independent of the parquet type system, making evolution easier. Downside
> > is loss of element-level encodings available today, though they can be
> > added later. Upside is simplicity and ease of evolution.
> >
> > - Option C1 - LIST with numeric parquet type elements as physical
> storage.
> > New float types (e.g. BF16) can be added in parquet proper as needed.
> > Downside is we need to add new types to parquet, DL/RL overhead is likely
> > not a concern. Upside is we have encoding support now.
>
> Option A is a short-term solution with severe limitations (lack of
> encoding choices) that make me entirely negative about the idea.
>
> I'm lukewarm about Option C (especially the optimistic statement that
> "DL/RL overhead is likely not a concern"), but at least it doesn't paint
> us in a corner.
>
> Regards
>
> Antoine.
>
>
>

Re: [DISCUSS] Numeric Vector type: physical representation (outcome of focused call)

[Will Edwards via dev <de...@parquet.apache.org>]

Hi all,

Andrew's and Gunnar's results show that existing LIST storage can be read
efficiently. This matches my own experience too. I think efficiency is a
software problem not a Parquet format problem.  E.g. the reader can expose
flat typed memory; nothing in the format requires allocating a List<Float>
per row. A nullable schema doesn't change that for pages without nulls,
which the reader can establish at page scope.

 So is the immediate question what SHOW CREATE TABLE should say when the
only source of schema is a Parquet file? ARRAY<FLOAT>, a fixed-length
array, or VECTOR<FLOAT, 768>? That's a useful discussion about the contract
we're expressing.

SELECT COS_SIM(a, b) etc can already work on numeric lists, with lengths
checked efficiently through the encoded levels. Whether a and b belong to
the same embedding space remains the user's responsibility; declaring both
to have 768 dimensions doesn't establish that.

There are other semantics I'd be interested in discussing:

  - Vector indexes, along the lines of Lance: how does a reader discover
that a column or auxiliary structure indexes another column, or is derived
from it? What lets an engine recognise when it can use that structure?

  - Quantisation, normalisation guarantees, and additional numeric formats
such as BF16/FP8. We already have FLOAT16 and small integer annotations,
but a coordinate type alone doesn't describe all of these.  Some quantised
representations are naturally composite values: packed coordinates plus
scales, offsets, or codebooks. Block-scaled FP4 is a concrete example.
These could be structs with separate children for codes and scales, and a
logical annotation describing how they collectively represent a vector.
Shared parameters might live elsewhere. Either way, the relationship needs
expressing.

Should we pin down those requirements before choosing a physical
representation?  Adding an element-type enum doesn't answer all of these,
and packing whole vectors into FLBA limits existing element encoding
choices.

 What concrete vector workload still needs a physical change after applying
the reader optimisations we've discussed?

 Will

On Tue, 8 Sept 2026 at 21:22, Antoine Pitrou <an...@python.org> wrote:

>
> Le 08/09/2026 à 20:42, Rok Mihevc a écrit :
> > Hi all,
> >
> > As proposed during the general community call we held a focused call
> today
> > to discuss the proposal for adding a logical vector type to parquet. This
> > was called because it seems:
> >
> > * Vectors (as in vector DB) storage needs are different to what we need
> for
> > fixed size list applications we've been disucssing so far [0].
>
> This seems unsubstantiated. Can you please provide an explanation?
>
> Ideally we'd use the *same* underlying solution for both problems.
>
> > We have converged on two options that could work, but have not yet
> decided
> > which one to propose:
> >
> > - Option A - FLBA as physical storage (FLBA(num_elements *
> element_width)).
> > Vector logical type would provide an enum specifying the element type,
> and
> > the element would be non-nullable. The element type enum could be
> > independent of the parquet type system, making evolution easier. Downside
> > is loss of element-level encodings available today, though they can be
> > added later. Upside is simplicity and ease of evolution.
> >
> > - Option C1 - LIST with numeric parquet type elements as physical
> storage.
> > New float types (e.g. BF16) can be added in parquet proper as needed.
> > Downside is we need to add new types to parquet, DL/RL overhead is likely
> > not a concern. Upside is we have encoding support now.
>
> Option A is a short-term solution with severe limitations (lack of
> encoding choices) that make me entirely negative about the idea.
>
> I'm lukewarm about Option C (especially the optimistic statement that
> "DL/RL overhead is likely not a concern"), but at least it doesn't paint
> us in a corner.
>
> Regards
>
> Antoine.
>
>
>

Re: [DISCUSS] Numeric Vector type: physical representation (outcome of focused call)

[Antoine Pitrou <an...@python.org>]

Le 08/09/2026 à 20:42, Rok Mihevc a écrit :
> Hi all,
> 
> As proposed during the general community call we held a focused call today
> to discuss the proposal for adding a logical vector type to parquet. This
> was called because it seems:
> 
> * Vectors (as in vector DB) storage needs are different to what we need for
> fixed size list applications we've been disucssing so far [0].

This seems unsubstantiated. Can you please provide an explanation?

Ideally we'd use the *same* underlying solution for both problems.

> We have converged on two options that could work, but have not yet decided
> which one to propose:
> 
> - Option A - FLBA as physical storage (FLBA(num_elements * element_width)).
> Vector logical type would provide an enum specifying the element type, and
> the element would be non-nullable. The element type enum could be
> independent of the parquet type system, making evolution easier. Downside
> is loss of element-level encodings available today, though they can be
> added later. Upside is simplicity and ease of evolution.
> 
> - Option C1 - LIST with numeric parquet type elements as physical storage.
> New float types (e.g. BF16) can be added in parquet proper as needed.
> Downside is we need to add new types to parquet, DL/RL overhead is likely
> not a concern. Upside is we have encoding support now.

Option A is a short-term solution with severe limitations (lack of 
encoding choices) that make me entirely negative about the idea.

I'm lukewarm about Option C (especially the optimistic statement that 
"DL/RL overhead is likely not a concern"), but at least it doesn't paint 
us in a corner.

Regards

Antoine.