You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "kaiwangleo (via GitHub)" <gi...@apache.org> on 2026/08/11 09:51:34 UTC

[PR] [hotfix][vector-store][java] Update Elasticsearch filters documentation [flink-agents]

kaiwangleo opened a new pull request, #997:
URL: https://github.com/apache/flink-agents/pull/997

   Linked issue: N/A (hotfix)
   
   ### Purpose of change
   
   Update stale Elasticsearch Vector Store Javadoc that incorrectly said the unified `filters` parameter was ignored.
   
   The documentation now describes the implemented behavior for `get`, `delete`, and `queryEmbedding`:
   
   - equality-only metadata matching;
   - mapping keys to `<metadataField>.<key>.keyword` term queries;
   - AND semantics across multiple entries;
   - AND composition with a raw JSON `filter_query`;
   - direct-ID behavior, where filters are not applied;
   - delete-all behavior when no IDs or filters are supplied.
   
   ### Tests
   
   - `git diff --check`
   - The changed `ElasticsearchVectorStore.java` passes Spotless formatting.
   - The module-level Spotless command continues to report only the unmodified `ElasticsearchVectorStoreTest.java`, whose checkout has CRLF line endings in the Windows-mounted workspace.
   
   No runtime tests were added because this is a documentation-only correction of behavior already covered by `testFiltersDsl`.
   
   ### API
   
   No public API or runtime behavior changes.
   
   ### Documentation
   
   - [ ] `doc-needed`
   - [ ] `doc-not-needed`
   - [x] `doc-included`
   
   ### Was this patch authored or co-authored using generative AI tooling?
   
   - [x] Yes
   - [ ] No
   
   Generated-by: OpenAI Codex Desktop 26.803.41515 (GPT-5.6 Sol)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [hotfix][vector-store][java] Update Elasticsearch filters documentation [flink-agents]

Posted by "weiqingy (via GitHub)" <gi...@apache.org>.
weiqingy commented on code in PR #997:
URL: https://github.com/apache/flink-agents/pull/997#discussion_r3837417384


##########
integrations/vector-stores/elasticsearch/src/main/java/org/apache/flink/agents/integrations/vectorstores/elasticsearch/ElasticsearchVectorStore.java:
##########
@@ -546,7 +554,7 @@ private void deleteDocuments(
         if (combined != null) {
             builder.query(q -> q.withJson(new StringReader(combined)));
         } else {
-            // No filter at all → delete every document (match_all).
+            // No filter at all 闂?delete every document (match_all).

Review Comment:
   Both checked out at `271818de` — nothing further from me on these two.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [hotfix][vector-store][java] Update Elasticsearch filters documentation [flink-agents]

Posted by "kaiwangleo (via GitHub)" <gi...@apache.org>.
kaiwangleo commented on code in PR #997:
URL: https://github.com/apache/flink-agents/pull/997#discussion_r3850580810


##########
integrations/vector-stores/elasticsearch/src/main/java/org/apache/flink/agents/integrations/vectorstores/elasticsearch/ElasticsearchVectorStore.java:
##########
@@ -336,23 +336,25 @@ public Map<String, Object> getStoreKwargs() {
     /**
      * Retrieve documents from the vector store.
      *
-     * <p>If ids is not provided, this method will retrieve documents according to {@code limit},
-     * {@code offset}, and {@code filter_query} in additional arguments. If {@code limit} is null,
-     * up to {@link ElasticsearchVectorStore#MAX_RESULT_WINDOW} documents are returned (an
-     * Elasticsearch ceiling).
+     * <p>When {@code ids} is non-empty, documents are retrieved directly by ID and the filter,
+     * limit, offset, and {@code filter_query} arguments are not applied.
      *
-     * <p>The unified {@code filters} DSL parameter is not yet translated to Elasticsearch's native
-     * query DSL — callers needing structured filtering should pass a raw {@code filter_query} via
-     * {@code extraArgs}. TODO: implement equality-DSL translation parallel to the Python Chroma
-     * implementation.
+     * <p>Otherwise, {@code filters} provides equality-only matching against document metadata. Each
+     * entry is translated to an Elasticsearch {@code term} query on {@code
+     * <metadataField>.<key>.keyword}, and multiple entries are combined with AND semantics. A raw
+     * Elasticsearch JSON query may also be supplied as {@code filter_query} in {@code extraArgs};
+     * when both forms are present, they are combined with AND semantics.
      *
-     * @param ids The ids of the documents.
-     * @param collection The name of the collection to be retrieved. If is null, retrieve the
-     *     default collection.
-     * @param filters Unified filter DSL. Currently ignored — see method Javadoc.
-     * @param limit Maximum number of documents to return; falls back to {@link
-     *     ElasticsearchVectorStore#MAX_RESULT_WINDOW} when null.
-     * @param extraArgs Additional arguments. (offset, filter_query, etc.)
+     * <p>The {@code limit} parameter takes precedence over a {@code limit} value in {@code
+     * extraArgs}. If neither is provided, up to {@link ElasticsearchVectorStore#MAX_RESULT_WINDOW}

Review Comment:
   Thanks for pointing out this distinction. `MAX_RESULT_WINDOW` is only the fallback request size used by this implementation when neither the `limit` parameter nor an `extraArgs` limit is provided. The effective `from + size` limit is controlled by the target index's `index.max_result_window` setting, which defaults to 10,000 but is configurable. I updated the `get` Javadoc and the constant comment accordingly in commit `608312b2`; no runtime behavior changed. The focused Spotless check and `git diff --check` pass.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [hotfix][vector-store][java] Update Elasticsearch filters documentation [flink-agents]

Posted by "kaiwangleo (via GitHub)" <gi...@apache.org>.
kaiwangleo commented on code in PR #997:
URL: https://github.com/apache/flink-agents/pull/997#discussion_r3793545051


##########
integrations/vector-stores/elasticsearch/src/main/java/org/apache/flink/agents/integrations/vectorstores/elasticsearch/ElasticsearchVectorStore.java:
##########
@@ -336,23 +336,29 @@ public Map<String, Object> getStoreKwargs() {
     /**
      * Retrieve documents from the vector store.
      *
-     * <p>If ids is not provided, this method will retrieve documents according to {@code limit},
-     * {@code offset}, and {@code filter_query} in additional arguments. If {@code limit} is null,
-     * up to {@link ElasticsearchVectorStore#MAX_RESULT_WINDOW} documents are returned (an
-     * Elasticsearch ceiling).
+     * <p>When {@code ids} is non-empty, documents are retrieved directly by ID and the filter,
+     * limit, offset, and {@code filter_query} arguments are not applied.
      *
-     * <p>The unified {@code filters} DSL parameter is not yet translated to Elasticsearch's native
-     * query DSL — callers needing structured filtering should pass a raw {@code filter_query} via
-     * {@code extraArgs}. TODO: implement equality-DSL translation parallel to the Python Chroma
-     * implementation.
+     * <p>Otherwise, {@code filters} provides equality-only matching against document metadata. Each
+     * entry is translated to an Elasticsearch {@code term} query on {@code
+     * <metadataField>.<key>.keyword}, and multiple entries are combined with AND semantics. Because
+     * Elasticsearch dynamic mapping creates `.keyword` sub-fields only for strings, filters on

Review Comment:
   
   
   All five occurrences now use Javadoc inline code tags: `{@code .keyword}` and `{@code filter_query}`. There are no Markdown backticks left in the file. 
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [hotfix][vector-store][java] Update Elasticsearch filters documentation [flink-agents]

Posted by "wenjin272 (via GitHub)" <gi...@apache.org>.
wenjin272 merged PR #997:
URL: https://github.com/apache/flink-agents/pull/997


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [hotfix][vector-store][java] Update Elasticsearch filters documentation [flink-agents]

Posted by "weiqingy (via GitHub)" <gi...@apache.org>.
weiqingy commented on code in PR #997:
URL: https://github.com/apache/flink-agents/pull/997#discussion_r3763613629


##########
integrations/vector-stores/elasticsearch/src/main/java/org/apache/flink/agents/integrations/vectorstores/elasticsearch/ElasticsearchVectorStore.java:
##########
@@ -571,17 +573,22 @@ private void deleteDocuments(
      * Executes a KNN vector search using a pre-computed embedding.
      *
      * <p>The method prepares a KNN search request using the supplied {@code embedding} and merges
-     * default arguments from the store with the provided {@code args}. Optional filter queries
-     * (JSON DSL) are applied as a post filter.
+     * default arguments from the store with the provided {@code args}. {@code filters} provides
+     * equality-only matching against metadata fields. Each entry targets {@code
+     * <metadataField>.<key>.keyword}; multiple entries are combined with AND semantics and applied
+     * as a post-filter.

Review Comment:
   Withdrawing the doc ask here. Rather than have you write down a limitation we intend to remove, I filed #999 for the pre-filter change and will take a first pass at the fix. Your post-filter sentence describes today's behavior accurately, so there is nothing to change in this PR for it.
   
   The other two comments are unaffected.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [hotfix][vector-store][java] Update Elasticsearch filters documentation [flink-agents]

Posted by "weiqingy (via GitHub)" <gi...@apache.org>.
weiqingy commented on code in PR #997:
URL: https://github.com/apache/flink-agents/pull/997#discussion_r3763532940


##########
integrations/vector-stores/elasticsearch/src/main/java/org/apache/flink/agents/integrations/vectorstores/elasticsearch/ElasticsearchVectorStore.java:
##########
@@ -336,23 +336,25 @@ public Map<String, Object> getStoreKwargs() {
     /**
      * Retrieve documents from the vector store.
      *
-     * <p>If ids is not provided, this method will retrieve documents according to {@code limit},
-     * {@code offset}, and {@code filter_query} in additional arguments. If {@code limit} is null,
-     * up to {@link ElasticsearchVectorStore#MAX_RESULT_WINDOW} documents are returned (an
-     * Elasticsearch ceiling).
+     * <p>When {@code ids} is non-empty, documents are retrieved directly by ID and the filter,
+     * limit, offset, and {@code filter_query} arguments are not applied.
      *
-     * <p>The unified {@code filters} DSL parameter is not yet translated to Elasticsearch's native
-     * query DSL — callers needing structured filtering should pass a raw {@code filter_query} via
-     * {@code extraArgs}. TODO: implement equality-DSL translation parallel to the Python Chroma
-     * implementation.
+     * <p>Otherwise, {@code filters} provides equality-only matching against document metadata. Each
+     * entry is translated to an Elasticsearch {@code term} query on {@code
+     * <metadataField>.<key>.keyword}, and multiple entries are combined with AND semantics. A raw
+     * Elasticsearch JSON query may also be supplied as {@code filter_query} in {@code extraArgs};
+     * when both forms are present, they are combined with AND semantics.
      *
-     * @param ids The ids of the documents.
-     * @param collection The name of the collection to be retrieved. If is null, retrieve the
-     *     default collection.
-     * @param filters Unified filter DSL. Currently ignored — see method Javadoc.
-     * @param limit Maximum number of documents to return; falls back to {@link
-     *     ElasticsearchVectorStore#MAX_RESULT_WINDOW} when null.
-     * @param extraArgs Additional arguments. (offset, filter_query, etc.)
+     * <p>The {@code limit} parameter takes precedence over a {@code limit} value in {@code
+     * extraArgs}. If neither is provided, up to {@link ElasticsearchVectorStore#MAX_RESULT_WINDOW}

Review Comment:
   nit: The text this replaces called `MAX_RESULT_WINDOW` an Elasticsearch ceiling, and the constant's own Javadoc still describes it as "the maximum number of documents that can be retrieved in get" (115). The new sentence reads more like a default that applies when `limit` is absent. Nothing clamps `size` (467), so someone who takes it that way and passes `limit = 50000` gets a rejection from Elasticsearch rather than 10000 rows.
   
   Is the ceiling sense worth keeping? Perhaps "…up to `MAX_RESULT_WINDOW` documents are returned; `offset` plus `limit` cannot exceed it either", though you may see a neater way to word it.



##########
integrations/vector-stores/elasticsearch/src/main/java/org/apache/flink/agents/integrations/vectorstores/elasticsearch/ElasticsearchVectorStore.java:
##########
@@ -336,23 +336,25 @@ public Map<String, Object> getStoreKwargs() {
     /**
      * Retrieve documents from the vector store.
      *
-     * <p>If ids is not provided, this method will retrieve documents according to {@code limit},
-     * {@code offset}, and {@code filter_query} in additional arguments. If {@code limit} is null,
-     * up to {@link ElasticsearchVectorStore#MAX_RESULT_WINDOW} documents are returned (an
-     * Elasticsearch ceiling).
+     * <p>When {@code ids} is non-empty, documents are retrieved directly by ID and the filter,
+     * limit, offset, and {@code filter_query} arguments are not applied.
      *
-     * <p>The unified {@code filters} DSL parameter is not yet translated to Elasticsearch's native
-     * query DSL — callers needing structured filtering should pass a raw {@code filter_query} via
-     * {@code extraArgs}. TODO: implement equality-DSL translation parallel to the Python Chroma
-     * implementation.
+     * <p>Otherwise, {@code filters} provides equality-only matching against document metadata. Each
+     * entry is translated to an Elasticsearch {@code term} query on {@code
+     * <metadataField>.<key>.keyword}, and multiple entries are combined with AND semantics. A raw

Review Comment:
   One detail from the helper's own Javadoc didn't make it up here: Elasticsearch only creates `.keyword` sub-fields for strings (776). The translation loop adds `.keyword` to every key whatever the value type (787-789), and metadata is stored as a plain object with dynamic mapping on (284, 294), so a number or boolean is mapped as `long`/`boolean` and never gets a `.keyword` at all. A `term` query against a field that isn't in the mapping doesn't error, it just matches nothing.
   
   The practical effect is that `Map.of("year", 2024)` comes back empty, with nothing to tell the caller the filter was never satisfiable. (Nothing alarming on `delete`, though. The helper still returns non-null, so the `match_all` branch at 551-552 stays untaken and a filter like that deletes nothing rather than everything.)
   
   Callers read this Javadoc rather than the private helper's, so would it be worth pulling that string-only qualifier up into it? Something like this, if it helps:
   
   > Because ES dynamic mapping only creates `.keyword` sub-fields for string values, filters on non-string metadata values will not match; use a raw `filter_query` for those.
   
   The same paragraph appears on `delete` (389-391) and `queryEmbedding` (576-578), so it would be three copies of the clause.



##########
integrations/vector-stores/elasticsearch/src/main/java/org/apache/flink/agents/integrations/vectorstores/elasticsearch/ElasticsearchVectorStore.java:
##########
@@ -571,17 +573,22 @@ private void deleteDocuments(
      * Executes a KNN vector search using a pre-computed embedding.
      *
      * <p>The method prepares a KNN search request using the supplied {@code embedding} and merges
-     * default arguments from the store with the provided {@code args}. Optional filter queries
-     * (JSON DSL) are applied as a post filter.
+     * default arguments from the store with the provided {@code args}. {@code filters} provides
+     * equality-only matching against metadata fields. Each entry targets {@code
+     * <metadataField>.<key>.keyword}; multiple entries are combined with AND semantics and applied
+     * as a post-filter.

Review Comment:
   Calling it a post-filter is accurate (623-625), and that word carries more weight than it might appear. Elasticsearch applies `post_filter` after the KNN phase has already chosen its `k` nearest hits, so it can only remove things from that set. It can't pull in matching documents that fell outside the top `k`.
   
   So `queryEmbedding(embedding, 5, coll, Map.of("user_id", "alice"), args)` can return nothing at all, even with hundreds of alice's documents indexed, if the five nearest vectors happen to belong to other people. That's different from `get` (476-481) and `delete` (542-549), where the same map becomes a real query clause and is exhaustive. The three paragraphs now read almost identically, which makes it easy to assume the behavior matches too.
   
   Would a clause noting that fewer than `k` documents can come back be worth adding? Whether the KNN clause's own `filter` option would suit this better than `post_filter` is an implementation question rather than a docs one, so I'll open a separate issue for that.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [hotfix][vector-store][java] Update Elasticsearch filters documentation [flink-agents]

Posted by "kaiwangleo (via GitHub)" <gi...@apache.org>.
kaiwangleo commented on code in PR #997:
URL: https://github.com/apache/flink-agents/pull/997#discussion_r3793545051


##########
integrations/vector-stores/elasticsearch/src/main/java/org/apache/flink/agents/integrations/vectorstores/elasticsearch/ElasticsearchVectorStore.java:
##########
@@ -336,23 +336,29 @@ public Map<String, Object> getStoreKwargs() {
     /**
      * Retrieve documents from the vector store.
      *
-     * <p>If ids is not provided, this method will retrieve documents according to {@code limit},
-     * {@code offset}, and {@code filter_query} in additional arguments. If {@code limit} is null,
-     * up to {@link ElasticsearchVectorStore#MAX_RESULT_WINDOW} documents are returned (an
-     * Elasticsearch ceiling).
+     * <p>When {@code ids} is non-empty, documents are retrieved directly by ID and the filter,
+     * limit, offset, and {@code filter_query} arguments are not applied.
      *
-     * <p>The unified {@code filters} DSL parameter is not yet translated to Elasticsearch's native
-     * query DSL — callers needing structured filtering should pass a raw {@code filter_query} via
-     * {@code extraArgs}. TODO: implement equality-DSL translation parallel to the Python Chroma
-     * implementation.
+     * <p>Otherwise, {@code filters} provides equality-only matching against document metadata. Each
+     * entry is translated to an Elasticsearch {@code term} query on {@code
+     * <metadataField>.<key>.keyword}, and multiple entries are combined with AND semantics. Because
+     * Elasticsearch dynamic mapping creates `.keyword` sub-fields only for strings, filters on

Review Comment:
   Fixed in `271818de` after merging the latest `upstream/main` in `ae5be353`.
   
   All five occurrences now use Javadoc inline code tags: `{@code .keyword}` and `{@code filter_query}`. There are no Markdown backticks left in the file. While rebuilding the Javadocs on the updated base, I also aligned `queryEmbedding` with the pre-filter behavior now present on `main` from #1001.
   
   Validation passed:
   
   - `bash tools/lint.sh -c`
   - `git diff --check`
   - the final diff against `upstream/main` contains only the three intended Javadoc sections
   



##########
integrations/vector-stores/elasticsearch/src/main/java/org/apache/flink/agents/integrations/vectorstores/elasticsearch/ElasticsearchVectorStore.java:
##########
@@ -546,7 +554,7 @@ private void deleteDocuments(
         if (combined != null) {
             builder.query(q -> q.withJson(new StringReader(combined)));
         } else {
-            // No filter at all → delete every document (match_all).
+            // No filter at all 闂?delete every document (match_all).

Review Comment:
   Fixed in `271818de` after merging the latest `upstream/main` in `ae5be353`.
   
   I resolved the conflicting Elasticsearch file from the exact upstream version before reapplying the three Javadoc updates. This preserves both original comments exactly:
   
   ```java
   // No filter at all → delete every document (match_all).
   // hit.score() is a Double — null for plain get-all responses, populated for
   ```
   
   The corrupted comment hunks are no longer present in the PR diff. The final diff against `upstream/main` contains only the `get`, `delete`, and `queryEmbedding` Javadocs. `bash tools/lint.sh -c` and `git diff --check` both pass.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [hotfix][vector-store][java] Update Elasticsearch filters documentation [flink-agents]

Posted by "wenjin272 (via GitHub)" <gi...@apache.org>.
wenjin272 commented on code in PR #997:
URL: https://github.com/apache/flink-agents/pull/997#discussion_r3828732550


##########
integrations/vector-stores/elasticsearch/src/main/java/org/apache/flink/agents/integrations/vectorstores/elasticsearch/ElasticsearchVectorStore.java:
##########
@@ -336,23 +336,25 @@ public Map<String, Object> getStoreKwargs() {
     /**
      * Retrieve documents from the vector store.
      *
-     * <p>If ids is not provided, this method will retrieve documents according to {@code limit},
-     * {@code offset}, and {@code filter_query} in additional arguments. If {@code limit} is null,
-     * up to {@link ElasticsearchVectorStore#MAX_RESULT_WINDOW} documents are returned (an
-     * Elasticsearch ceiling).
+     * <p>When {@code ids} is non-empty, documents are retrieved directly by ID and the filter,
+     * limit, offset, and {@code filter_query} arguments are not applied.
      *
-     * <p>The unified {@code filters} DSL parameter is not yet translated to Elasticsearch's native
-     * query DSL — callers needing structured filtering should pass a raw {@code filter_query} via
-     * {@code extraArgs}. TODO: implement equality-DSL translation parallel to the Python Chroma
-     * implementation.
+     * <p>Otherwise, {@code filters} provides equality-only matching against document metadata. Each
+     * entry is translated to an Elasticsearch {@code term} query on {@code
+     * <metadataField>.<key>.keyword}, and multiple entries are combined with AND semantics. A raw
+     * Elasticsearch JSON query may also be supplied as {@code filter_query} in {@code extraArgs};
+     * when both forms are present, they are combined with AND semantics.
      *
-     * @param ids The ids of the documents.
-     * @param collection The name of the collection to be retrieved. If is null, retrieve the
-     *     default collection.
-     * @param filters Unified filter DSL. Currently ignored — see method Javadoc.
-     * @param limit Maximum number of documents to return; falls back to {@link
-     *     ElasticsearchVectorStore#MAX_RESULT_WINDOW} when null.
-     * @param extraArgs Additional arguments. (offset, filter_query, etc.)
+     * <p>The {@code limit} parameter takes precedence over a {@code limit} value in {@code
+     * extraArgs}. If neither is provided, up to {@link ElasticsearchVectorStore#MAX_RESULT_WINDOW}

Review Comment:
   One nuance here: `MAX_RESULT_WINDOW` is only the request-size fallback used by this implementation, while the actual result-window limit is controlled by the per-index `index.max_result_window` setting. `10000` is only its default value.
   
   If the index setting is raised, a limit above `10000` may succeed; if it is lowered, even the default request size may fail. Could we clarify this distinction? For example:
   
   > If neither is provided, the request size defaults to `MAX_RESULT_WINDOW`. Elasticsearch rejects requests whose offset plus limit exceeds the index’s `index.max_result_window` setting, which defaults to 10,000.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [hotfix][vector-store][java] Update Elasticsearch filters documentation [flink-agents]

Posted by "weiqingy (via GitHub)" <gi...@apache.org>.
weiqingy commented on code in PR #997:
URL: https://github.com/apache/flink-agents/pull/997#discussion_r3780489730


##########
integrations/vector-stores/elasticsearch/src/main/java/org/apache/flink/agents/integrations/vectorstores/elasticsearch/ElasticsearchVectorStore.java:
##########
@@ -336,23 +336,29 @@ public Map<String, Object> getStoreKwargs() {
     /**
      * Retrieve documents from the vector store.
      *
-     * <p>If ids is not provided, this method will retrieve documents according to {@code limit},
-     * {@code offset}, and {@code filter_query} in additional arguments. If {@code limit} is null,
-     * up to {@link ElasticsearchVectorStore#MAX_RESULT_WINDOW} documents are returned (an
-     * Elasticsearch ceiling).
+     * <p>When {@code ids} is non-empty, documents are retrieved directly by ID and the filter,
+     * limit, offset, and {@code filter_query} arguments are not applied.
      *
-     * <p>The unified {@code filters} DSL parameter is not yet translated to Elasticsearch's native
-     * query DSL — callers needing structured filtering should pass a raw {@code filter_query} via
-     * {@code extraArgs}. TODO: implement equality-DSL translation parallel to the Python Chroma
-     * implementation.
+     * <p>Otherwise, {@code filters} provides equality-only matching against document metadata. Each
+     * entry is translated to an Elasticsearch {@code term} query on {@code
+     * <metadataField>.<key>.keyword}, and multiple entries are combined with AND semantics. Because
+     * Elasticsearch dynamic mapping creates `.keyword` sub-fields only for strings, filters on

Review Comment:
   Thanks for addressing comments.
   
   nit: The new sentences use markdown backticks for `` `.keyword` `` and `` `filter_query` ``, but Javadoc has no markdown, so those render as literal backtick characters in the generated docs. The file uses `{@code ...}` 71 times elsewhere, and `main` has no backticks at all.
   
   The `queryEmbedding` copy shows the split within one sentence: `{@code filter_query}` at 586 sits next to `` `.keyword` `` at 585.
   
   Would you switch the five occurrences (345, 346, 396, 397, 585) over to `{@code ...}`?



##########
integrations/vector-stores/elasticsearch/src/main/java/org/apache/flink/agents/integrations/vectorstores/elasticsearch/ElasticsearchVectorStore.java:
##########
@@ -546,7 +554,7 @@ private void deleteDocuments(
         if (combined != null) {
             builder.query(q -> q.withJson(new StringReader(combined)));
         } else {
-            // No filter at all → delete every document (match_all).
+            // No filter at all 闂?delete every document (match_all).

Review Comment:
   At head this line reads:
   
   ```
   // No filter at all 闂?delete every document (match_all).
   ```
   
   On `main` it is `// No filter at all → delete every document (match_all).` The arrow has turned into `闂` followed by a literal `?` (bytes `e9 97 82 3f`). Line 736 has the same corruption, where an em dash became the identical sequence: `Double 闂?null`.
   
   I bisected it to `41c78db8`, so it isn't fallout from the KNN commits. Neither line is documentation, and both now show up as extra hunks. CI is green on this, so nothing catches it automatically.
   
   Since restoring them would also bring the diff back to just the three Javadoc hunks, could you put both lines back to what `main` has?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [hotfix][vector-store][java] Update Elasticsearch filters documentation [flink-agents]

Posted by "kaiwangleo (via GitHub)" <gi...@apache.org>.
kaiwangleo commented on code in PR #997:
URL: https://github.com/apache/flink-agents/pull/997#discussion_r3764207664


##########
integrations/vector-stores/elasticsearch/src/main/java/org/apache/flink/agents/integrations/vectorstores/elasticsearch/ElasticsearchVectorStore.java:
##########
@@ -336,23 +336,25 @@ public Map<String, Object> getStoreKwargs() {
     /**
      * Retrieve documents from the vector store.
      *
-     * <p>If ids is not provided, this method will retrieve documents according to {@code limit},
-     * {@code offset}, and {@code filter_query} in additional arguments. If {@code limit} is null,
-     * up to {@link ElasticsearchVectorStore#MAX_RESULT_WINDOW} documents are returned (an
-     * Elasticsearch ceiling).
+     * <p>When {@code ids} is non-empty, documents are retrieved directly by ID and the filter,
+     * limit, offset, and {@code filter_query} arguments are not applied.
      *
-     * <p>The unified {@code filters} DSL parameter is not yet translated to Elasticsearch's native
-     * query DSL — callers needing structured filtering should pass a raw {@code filter_query} via
-     * {@code extraArgs}. TODO: implement equality-DSL translation parallel to the Python Chroma
-     * implementation.
+     * <p>Otherwise, {@code filters} provides equality-only matching against document metadata. Each
+     * entry is translated to an Elasticsearch {@code term} query on {@code
+     * <metadataField>.<key>.keyword}, and multiple entries are combined with AND semantics. A raw

Review Comment:
   Thanks for checking the helper implementation in detail. You are right: Elasticsearch dynamic mapping creates .keyword sub-fields only for string metadata values, while the equality DSL always targets <metadataField>.<key>.keyword. I added the limitation and the raw ilter_query workaround to the get, delete, and queryEmbedding Javadocs in commit 41c78db8. This documents the current non-string behavior without changing runtime semantics.



##########
integrations/vector-stores/elasticsearch/src/main/java/org/apache/flink/agents/integrations/vectorstores/elasticsearch/ElasticsearchVectorStore.java:
##########
@@ -571,17 +573,22 @@ private void deleteDocuments(
      * Executes a KNN vector search using a pre-computed embedding.
      *
      * <p>The method prepares a KNN search request using the supplied {@code embedding} and merges
-     * default arguments from the store with the provided {@code args}. Optional filter queries
-     * (JSON DSL) are applied as a post filter.
+     * default arguments from the store with the provided {@code args}. {@code filters} provides
+     * equality-only matching against metadata fields. Each entry targets {@code
+     * <metadataField>.<key>.keyword}; multiple entries are combined with AND semantics and applied
+     * as a post-filter.

Review Comment:
   Thanks for the clarification and for opening issue #999. I agree the current post-filter wording accurately describes today's behavior, so I left that part unchanged in this PR. The two remaining documentation clarifications are included in commit 41c78db8.



##########
integrations/vector-stores/elasticsearch/src/main/java/org/apache/flink/agents/integrations/vectorstores/elasticsearch/ElasticsearchVectorStore.java:
##########
@@ -336,23 +336,25 @@ public Map<String, Object> getStoreKwargs() {
     /**
      * Retrieve documents from the vector store.
      *
-     * <p>If ids is not provided, this method will retrieve documents according to {@code limit},
-     * {@code offset}, and {@code filter_query} in additional arguments. If {@code limit} is null,
-     * up to {@link ElasticsearchVectorStore#MAX_RESULT_WINDOW} documents are returned (an
-     * Elasticsearch ceiling).
+     * <p>When {@code ids} is non-empty, documents are retrieved directly by ID and the filter,
+     * limit, offset, and {@code filter_query} arguments are not applied.
      *
-     * <p>The unified {@code filters} DSL parameter is not yet translated to Elasticsearch's native
-     * query DSL — callers needing structured filtering should pass a raw {@code filter_query} via
-     * {@code extraArgs}. TODO: implement equality-DSL translation parallel to the Python Chroma
-     * implementation.
+     * <p>Otherwise, {@code filters} provides equality-only matching against document metadata. Each
+     * entry is translated to an Elasticsearch {@code term} query on {@code
+     * <metadataField>.<key>.keyword}, and multiple entries are combined with AND semantics. A raw
+     * Elasticsearch JSON query may also be supplied as {@code filter_query} in {@code extraArgs};
+     * when both forms are present, they are combined with AND semantics.
      *
-     * @param ids The ids of the documents.
-     * @param collection The name of the collection to be retrieved. If is null, retrieve the
-     *     default collection.
-     * @param filters Unified filter DSL. Currently ignored — see method Javadoc.
-     * @param limit Maximum number of documents to return; falls back to {@link
-     *     ElasticsearchVectorStore#MAX_RESULT_WINDOW} when null.
-     * @param extraArgs Additional arguments. (offset, filter_query, etc.)
+     * <p>The {@code limit} parameter takes precedence over a {@code limit} value in {@code
+     * extraArgs}. If neither is provided, up to {@link ElasticsearchVectorStore#MAX_RESULT_WINDOW}

Review Comment:
   Good point. I restored the ceiling wording and clarified that MAX_RESULT_WINDOW is the Elasticsearch result-window limit, not an automatic clamp. The Javadoc now states that the combined offset and limit must not exceed the window and that an explicit limit above it is rejected by Elasticsearch rather than truncated. Updated in commit 41c78db8.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [hotfix][vector-store][java] Update Elasticsearch filters documentation [flink-agents]

Posted by "wenjin272 (via GitHub)" <gi...@apache.org>.
wenjin272 commented on PR #997:
URL: https://github.com/apache/flink-agents/pull/997#issuecomment-5407761582

   Thanks for addressing the comments. LGTM


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org