You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "steFaiz (via GitHub)" <gi...@apache.org> on 2026/06/18 15:59:35 UTC
[PR] [parquet] add vector support for parquet [paimon]
steFaiz opened a new pull request, #8282:
URL: https://github.com/apache/paimon/pull/8282
### Purpose
Currently parquet format do not recognize vector type. This will cause `UnsupportedOperationException` if users' do not store vectors in separated format. e.g. Lance, Vortex
### Tests
UnitTests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [parquet] add vector support for parquet [paimon]
Posted by "steFaiz (via GitHub)" <gi...@apache.org>.
steFaiz commented on PR #8282:
URL: https://github.com/apache/paimon/pull/8282#issuecomment-4762370322
Thanks for your review! Fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [parquet] add vector support for parquet [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #8282:
URL: https://github.com/apache/paimon/pull/8282#issuecomment-4747256058
Thanks for adding Parquet vector support. One correctness concern: VectorType is fixed-size, but this implementation encodes it as a Parquet LIST and currently accepts whatever list length is present. On the write path, ParquetRowDataWriter writes row.getVector(...).size() without checking it against VectorType.getLength(); on the read path, CastedVectorColumnVector returns a ColumnarVec using the Parquet list length, not the declared vector length. That can let malformed vectors be written/read and then fail later in Spark/DataConverter or vector indexing with less context. Could we validate the vector length at the Parquet boundary, and ideally add a test for mismatched length?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [parquet] add vector support for parquet [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #8282:
URL: https://github.com/apache/paimon/pull/8282
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [parquet] add vector support for parquet [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #8282:
URL: https://github.com/apache/paimon/pull/8282#issuecomment-4766303084
+1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org