You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "YAshhh29 (via GitHub)" <gi...@apache.org> on 2026/10/09 21:23:08 UTC

[PR] Add QdrantSearchOperator for vector similarity search [airflow]

YAshhh29 opened a new pull request, #74521:
URL: https://github.com/apache/airflow/pull/74521

   Replaces #69673, which was closed under the open-PR limit and can't be reopened from my side. Same commits, rebased onto main with no conflicts.
   
   The Qdrant provider can write vectors with `QdrantIngestOperator` but has no operator to read them back, so a similarity search from a Dag means calling `hook.conn.query_points` directly and converting the returned `ScoredPoint` objects to dicts by hand before they can go to XCom.
   
   - `QdrantHook.search()` wraps `QdrantClient.query_points` (not the deprecated `search()`) and returns `list[dict]`, so results are XCom-safe. Extra keyword arguments are forwarded, which covers named vectors and hybrid search.
   - `QdrantSearchOperator` is a thin operator on top. `conn_id`, `collection_name`, `query`, `query_filter` and `limit` are templated, so a query vector can come from an upstream embedding task.
   - The example Dag and the how-to guide gain a search task after the ingest task.
   
   Unit tests cover the hook and the operator, and pass against `qdrant-client` 1.19.1.
   
   
   ---
   
   ##### Was generative AI tooling used to co-author this PR?
   
   - [X] Yes. I used an AI coding assistant (GitHub Copilot, Claude Code) for parts of the change and for the rebase. I reviewed everything before pushing and I'm responsible for it.
   
   Generated-by: Claude Code following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add QdrantSearchOperator for vector similarity search [airflow]

Posted by "YAshhh29 (via GitHub)" <gi...@apache.org>.
YAshhh29 commented on PR #74521:
URL: https://github.com/apache/airflow/pull/74521#issuecomment-6089497431

   Reopening this now that I'm under the open-PR limit. Rebased onto main with no conflicts; the Qdrant unit tests pass against `qdrant-client` 1.19.1 and `conn_id` is templated as #73286 requires. Would appreciate a review when someone has time.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add QdrantSearchOperator for vector similarity search [airflow]

Posted by "YAshhh29 (via GitHub)" <gi...@apache.org>.
YAshhh29 commented on PR #74521:
URL: https://github.com/apache/airflow/pull/74521#issuecomment-6385234739

   Ran both operators against Qdrant's in-process engine (`QdrantClient(":memory:")`): ingest then search, filters, score threshold, offset, named vectors through `query_points_kwargs`, and sparse vectors all work, and results come back as plain JSON for XCom. Two of these are now tests, so CI checks the real engine too.
   
   <img width="1822" height="725" alt="download" src="https://github.com/user-attachments/assets/25bc3d2a-b367-4775-80e6-4f599f59f84b" />
   
   The workflows for the latest push are waiting for approval, if a maintainer could approve them.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org