You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@fluss.apache.org by "vamossagar12 (via GitHub)" <gi...@apache.org> on 2026/03/16 05:35:23 UTC
[D] Some questions on LanceDB Lakehouse Integration [fluss]
GitHub user vamossagar12 created a discussion: Some questions on LanceDB Lakehouse Integration
I was looking at the LanceDB [lakehouse integration](https://fluss.apache.org/docs/streaming-lakehouse/integrate-data-lakes/lance/#vector-embedding-example) and had some questions on the same.
1) I noticed that when creating a table in Fluss, the vector index type is not configurable. Based on my understanding, LanceDB supports [3 types of indexes](https://docs.lancedb.com/indexing/vector-index#index-tuning), so I was wondering if it should be possible to configure this via Fluss? I think having separate index type is useful to allow creating indexes to cater to multiple usecases/embedding types. LanceDB has 2 separate steps for creating a table and creating an index on top of it. Is the expectation that users would use it to create only a table via Fluss and have index created separately? That could be a friction for users.
2) A similar question arises for hybrid search which is also supported by LanceDB. That might need configuring a different set of indexes for FTS described [here](https://docs.lancedb.com/search/full-text-search#tokenize-table-data). Does Fluss look at also enabling hybrid search via the index it sets up?
3) Another question is about [multivector support](https://docs.lancedb.com/search/multivector-search#multivector-support). I think LanceDB expects such vectors to be stored in lists of lists but that is a data type not supported in Fluss I believe. Maybe Fluss can model it as users creating a table with <id> and different columns or rows with vectors and Fluss internally does the mapping of conerting it to the above format. I think this is also an important area so maybe a Fluss integrated solution could be useful.
Let me know your thoughts!
GitHub link: https://github.com/apache/fluss/discussions/2877
----
This is an automatically sent email for issues@fluss.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@fluss.apache.org
Re: [D] Some questions on LanceDB Lakehouse Integration [fluss]
Posted by "vamossagar12 (via GitHub)" <gi...@apache.org>.
GitHub user vamossagar12 added a comment to the discussion: Some questions on LanceDB Lakehouse Integration
Thanks @polyzos ! What is the process of introducing such new features? Do we need to write some proposal docs?
GitHub link: https://github.com/apache/fluss/discussions/2877#discussioncomment-16201734
----
This is an automatically sent email for issues@fluss.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@fluss.apache.org
Re: [D] Some questions on LanceDB Lakehouse Integration [fluss]
Posted by "vamossagar12 (via GitHub)" <gi...@apache.org>.
GitHub user vamossagar12 added a comment to the discussion: Some questions on LanceDB Lakehouse Integration
Ok, maybe a proposal would be better given this would be a new capability in Fluss - to create indexes etc along with new set of configs.
GitHub link: https://github.com/apache/fluss/discussions/2877#discussioncomment-16288042
----
This is an automatically sent email for issues@fluss.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@fluss.apache.org
Re: [D] Some questions on LanceDB Lakehouse Integration [fluss]
Posted by "polyzos (via GitHub)" <gi...@apache.org>.
GitHub user polyzos added a comment to the discussion: Some questions on LanceDB Lakehouse Integration
@vamossagar12 You're correct that vector index creation is not currently configurable through Fluss. The LanceDB integration today is a storage/tiering layer and it handles writing Fluss log records into Lance datasets and atomic fragment appends, but does not invoke any of LanceDB's index creation APIs.
As it stands, users would need to create indexes separately through LanceDB's API after the table is created via Fluss. I agree this creates friction. A natural improvement would be to expose index configuration through table properties (e.g., `lance.index.type`, `lance.index.metric`, `lance.index.num_partitions`) and have Fluss call LanceDB's index creation API as part of table creation. This is something we should track for improvement.
The other two are also not supported.. In general we should strengthen more our vector/lance support and we need to keep track of what can be improved and we welcome new idea, so thanks all lot for sharing all the above.
GitHub link: https://github.com/apache/fluss/discussions/2877#discussioncomment-16184232
----
This is an automatically sent email for issues@fluss.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@fluss.apache.org
Re: [D] Some questions on LanceDB Lakehouse Integration [fluss]
Posted by "polyzos (via GitHub)" <gi...@apache.org>.
GitHub user polyzos deleted a comment on the discussion: Some questions on LanceDB Lakehouse Integration
It depends on the feature and impact.
For big features it’s best to start with a proposal
GitHub link: https://github.com/apache/fluss/discussions/2877#discussioncomment-16232712
----
This is an automatically sent email for issues@fluss.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@fluss.apache.org
Re: [D] Some questions on LanceDB Lakehouse Integration [fluss]
Posted by "polyzos (via GitHub)" <gi...@apache.org>.
GitHub user polyzos added a comment to the discussion: Some questions on LanceDB Lakehouse Integration
It depends on the feature and impact.
For big features it’s best to start with a proposal
GitHub link: https://github.com/apache/fluss/discussions/2877#discussioncomment-16232712
----
This is an automatically sent email for issues@fluss.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@fluss.apache.org
Re: [D] Some questions on LanceDB Lakehouse Integration [fluss]
Posted by "vamossagar12 (via GitHub)" <gi...@apache.org>.
GitHub user vamossagar12 added a comment to the discussion: Some questions on LanceDB Lakehouse Integration
Tagging @wuchong , @polyzos , @luoyuxia for more reviews.
GitHub link: https://github.com/apache/fluss/discussions/2877#discussioncomment-16172332
----
This is an automatically sent email for issues@fluss.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@fluss.apache.org
Re: [D] Some questions on LanceDB Lakehouse Integration [fluss]
Posted by "polyzos (via GitHub)" <gi...@apache.org>.
GitHub user polyzos added a comment to the discussion: Some questions on LanceDB Lakehouse Integration
It depends on the feature and impact.
For big features it’s best to start with a proposal
On Thu, 19 Mar 2026 at 5:42 AM, vamossagar12 ***@***.***>
wrote:
> Thanks @polyzos <https://github.com/polyzos> ! What is the process of
> introducing such new features? Do we need to write some proposal docs?
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/apache/fluss/discussions/2877#discussioncomment-16201734>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AFTW3PNV3BABREPEMOZA2BT4RNUCVAVCNFSM6AAAAACWS55IYSVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTMMRQGE3TGNA>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>
GitHub link: https://github.com/apache/fluss/discussions/2877#discussioncomment-16232711
----
This is an automatically sent email for issues@fluss.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@fluss.apache.org