You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "cshuo (via GitHub)" <gi...@apache.org> on 2026/05/08 09:32:55 UTC

Re: [I] Add Flink support for HoodieSchema Vector type [hudi]

cshuo commented on issue #18506:
URL: https://github.com/apache/hudi/issues/18506#issuecomment-4405337541

   Thanks for opening this. I think we can split this into a few pieces.
   
   1. First, for the Flink user API, Flink currently does not have a native `VECTOR` type, and we probably cannot mirror Spark SQL exactly by adding a new `VECTOR` type in DDL, since the Flink SQL parser is not as flexible for type extensions. One feasible approach is to declare the column as an array in Flink SQL and specify the Hudi vector columns through table options, for example:
   ```
       CREATE TABLE t (
         id BIGINT,
         embedding1 ARRAY<FLOAT> NOT NULL,
         embedding2 ARRAY<DOUBLE> NOT NULL,
         PRIMARY KEY (id) NOT ENFORCED
       ) WITH (
         'connector' = 'hudi',
         'path' = '...',
         'hoodie.vector.columns' = 'embedding1:VECTOR(128), embedding2:VECTOR(64)'
       );
   ```
   This keeps the Flink schema valid while giving Hudi enough metadata to treat `embedding` as a `VECTOR(128)` column.
   
   2. Then, the Flink read/write path should support Hudi `VECTOR` columns. For example, if Spark creates a Hudi table with a vector column, Flink should be able to read and write that table correctly. On the Flink side, the physical/user-visible type can still be `ARRAY<FLOAT>` / `ARRAY<DOUBLE>` / `ARRAY<TINYINT>`, while Hudi preserves the vector descriptor, dimension, and storage semantics internally.
   
   3. Flink `VECTOR_SEARCH` support can be handled as a follow-up. Flink’s `VECTOR_SEARCH` works on `FLOAT ARRAY` / `DOUBLE ARRAY` columns directly, so it does not fundamentally depend on Hudi’s `VECTOR` logical type. Once Hudi vector columns round-trip as Flink arrays, users should already be able to apply Flink’s vector search API on top of them; any deeper integration or optimization can be tracked separately.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org