You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "twosom (via GitHub)" <gi...@apache.org> on 2025/12/26 03:18:05 UTC

[PR] doc : add ElasticsearchVectorStore doc [flink-agents]

twosom opened a new pull request, #397:
URL: https://github.com/apache/flink-agents/pull/397

   <!--
   * Thank you very much for contributing to Flink Agents.
   * Please add the relevant components in the PR title. E.g., [api], [runtime], [java], [python], [hotfix], etc.
   -->
   
   <!-- Please link the PR to the relevant issue(s). Hotfix doesn't need this. -->
   Linked issue: #385, #321
   
   ### Purpose of change
   
   <!-- What is the purpose of this change? -->
   Added documentation for ElasticsearchVectorStore.
   
   
   ### Tests
   
   <!-- How is this change verified? -->
   
   ### API
   
   <!-- Does this change touches any public APIs? -->
   
   ### Documentation
   
   <!-- Do not remove this section. Check the proper box only. -->
   
   - [ ] `doc-needed` <!-- Your PR changes impact docs -->
   - [ ] `doc-not-needed` <!-- Your PR changes do not impact docs -->
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] doc : add ElasticsearchVectorStore doc [flink-agents]

Posted by "xintongsong (via GitHub)" <gi...@apache.org>.
xintongsong merged PR #397:
URL: https://github.com/apache/flink-agents/pull/397


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] doc : add ElasticsearchVectorStore doc [flink-agents]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #397:
URL: https://github.com/apache/flink-agents/pull/397#issuecomment-3691993034

   @twosom Please add the following content to your PR description and select a checkbox:
   ```
   - [ ] `doc-needed` 
   - [ ] `doc-not-needed` <!-- Your PR changes do not impact docs -->
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] doc : add ElasticsearchVectorStore doc [flink-agents]

Posted by "xintongsong (via GitHub)" <gi...@apache.org>.
xintongsong commented on code in PR #397:
URL: https://github.com/apache/flink-agents/pull/397#discussion_r2674492384


##########
docs/content/docs/development/vector_stores.md:
##########
@@ -67,26 +83,64 @@ query = VectorStoreQuery(
 )
 ```
 
-### Query Results
+{{< /tab >}}
 
-When you execute a query, you receive a `VectorStoreQueryResult` object that contains the search results:
+{{< tab "Java" >}}
 
-```python
-# Execute the query
-result = vector_store.query(query)
+```java
+// Create a semantic search query
+VectorStoreQuery query = new VectorStoreQuery(
+        VectorStoreQueryMode.SEMANTIC,
+        "What is Apache Flink Agents?",
+        3
+);

Review Comment:
   Currently, `SEMANTIC` is the only supported mode. That's why `KEYWORD` & `HYBRID` are commented out in Python. I think in Java we probably should also hide the modes from user for the moment. That means, we can also comment out the unsupported modes, and use the constructor without the mode argument in the document. WDYT?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] doc : add ElasticsearchVectorStore doc [flink-agents]

Posted by "xintongsong (via GitHub)" <gi...@apache.org>.
xintongsong commented on code in PR #397:
URL: https://github.com/apache/flink-agents/pull/397#discussion_r2666768814


##########
docs/content/docs/development/vector_stores.md:
##########
@@ -67,26 +83,64 @@ query = VectorStoreQuery(
 )
 ```
 
-### Query Results
+{{< /tab >}}
 
-When you execute a query, you receive a `VectorStoreQueryResult` object that contains the search results:
+{{< tab "Java" >}}
 
-```python
-# Execute the query
-result = vector_store.query(query)
+```java
+// Create a semantic search query
+VectorStoreQuery query = new VectorStoreQuery(
+        VectorStoreQueryMode.SEMANTIC,
+        "What is Apache Flink Agents?",
+        3
+);

Review Comment:
   Unlike in python where we have keyword arguments, in java we'd better have some comments to help users understand the arguments, especially when the arguments are literals rather than variables with meaningful names.
   ```suggestion
   VectorStoreQuery query = new VectorStoreQuery(
           VectorStoreQueryMode.SEMANTIC, // mode
           "What is Apache Flink Agents?", // query text
           3 // limit
   );
   ```



##########
docs/content/docs/development/vector_stores.md:
##########
@@ -153,6 +268,10 @@ class MyAgent(Agent):
 
 #### ChromaVectorStore Parameters
 
+{{< tabs "ChromaVectorStore Parameters" >}}
+
+{{< tab "Python" >}}

Review Comment:
   We probably don't need tabs for the parameters. Even later we support Chroma in Java, I think the parameters should be language-independent.



##########
docs/content/docs/development/vector_stores.md:
##########
@@ -265,6 +396,74 @@ def chroma_store() -> ResourceDescriptor:
     )
 ```
 
+### Elasticsearch
+
+[Elasticsearch](https://www.elastic.co/elasticsearch/) is a distributed, RESTful search and analytics engine that supports vector search through dense vector fields and K-Nearest Neighbors (KNN).
+
+{{< hint info >}}
+Elasticsearch is currently supported in the Java API only.
+{{< /hint >}}
+
+#### Prerequisites
+
+1. An Elasticsearch cluster (version 8.0 or later for KNN support).
+2. An index with a `dense_vector` field.
+
+#### ElasticsearchVectorStore Parameters
+
+{{< tabs "ElasticsearchVectorStore Parameters" >}}
+
+{{< tab "Java" >}}

Review Comment:
   Same here for the parameter tabs.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] doc : add ElasticsearchVectorStore doc [flink-agents]

Posted by "xintongsong (via GitHub)" <gi...@apache.org>.
xintongsong commented on code in PR #397:
URL: https://github.com/apache/flink-agents/pull/397#discussion_r2648991844


##########
docs/content/docs/development/vector_stores.md:
##########
@@ -265,6 +285,105 @@ def chroma_store() -> ResourceDescriptor:
     )
 ```
 
+### Elasticsearch
+
+[Elasticsearch](https://www.elastic.co/elasticsearch/) is a distributed, RESTful search and analytics engine that supports vector search through dense vector fields and K-Nearest Neighbors (KNN).
+
+{{< hint info >}}
+Elasticsearch is currently supported in the Java API only.
+{{< /hint >}}
+
+#### Prerequisites
+
+1. An Elasticsearch cluster (version 8.0 or later for KNN support).
+2. An index with a `dense_vector` field.
+
+#### ElasticsearchVectorStore Parameters
+
+{{< tabs "ElasticsearchVectorStore Parameters" >}}
+
+{{< tab "Java" >}}
+
+| Parameter | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `embedding_model` | str | Required | Reference to embedding model resource name |
+| `index` | str | Required | Target Elasticsearch index name |
+| `vector_field` | str | Required | Name of the dense vector field used for KNN |
+| `dims` | int | `768` | Vector dimensionality |
+| `k` | int | None | Number of nearest neighbors to return; can be overridden per query |
+| `num_candidates` | int | None | Candidate set size for ANN search; can be overridden per query |
+| `filter_query` | str | None | Raw JSON Elasticsearch filter query (DSL) applied as a post-filter |
+| `host` | str | `"http://localhost:9200"` | Elasticsearch endpoint |
+| `hosts` | str | None | Comma-separated list of Elasticsearch endpoints |
+| `username` | str | None | Username for basic authentication |
+| `password` | str | None | Password for basic authentication |
+| `api_key_base64` | str | None | Base64-encoded API key for authentication |
+| `api_key_id` | str | None | API key ID for authentication |
+| `api_key_secret` | str | None | API key secret for authentication |
+
+{{< /tab >}}
+
+{{< /tabs >}}
+
+#### Usage Example
+
+{{< tabs "Elasticsearch Usage Example" >}}
+
+{{< tab "Java" >}}
+
+Here's how to define an Elasticsearch vector store in your Java agent:
+
+```java
+public class MyAgent extends Agent {
+
+    @EmbeddingModelConnection
+    public static ResourceDescriptor embeddingConnection() {
+        return ResourceDescriptor.Builder.newBuilder(OpenAIEmbeddingModelConnection.class.getName())
+                .addInitialArgument("api_key", "your-api-key-here")
+                .build();
+    }
+
+    @EmbeddingModelSetup
+    public static ResourceDescriptor embeddingModel() {
+        return ResourceDescriptor.Builder.newBuilder(OpenAIEmbeddingModelSetup.class.getName())
+                .addInitialArgument("connection", "embeddingConnection")
+                .addInitialArgument("model", "text-embedding-3-small")
+                .build();
+    }
+
+    @VectorStore
+    public static ResourceDescriptor vectorStore() {
+        return ResourceDescriptor.Builder.newBuilder(ElasticsearchVectorStore.class.getName())
+                .addInitialArgument("embedding_model", "embeddingModel")
+                .addInitialArgument("host", "http://localhost:9200")
+                .addInitialArgument("index", "my_documents")
+                .addInitialArgument("vector_field", "content_vector")
+                .addInitialArgument("dims", 1536)
+                // Optional authentication
+                // .addInitialArgument("username", "elastic")
+                // .addInitialArgument("password", "secret")
+                .build();
+    }
+
+    @Action(listenEvents = InputEvent.class)
+    public static void searchDocuments(InputEvent event, RunnerContext ctx) {
+        String query = (String) event.getInput();
+        // Request context retrieval via the vector store
+        ctx.sendEvent(new ContextRetrievalRequestEvent(query, "vectorStore"));
+    }
+
+    @Action(listenEvents = ContextRetrievalResponseEvent.class)
+    public static void onSearchResponse(ContextRetrievalResponseEvent event, RunnerContext ctx) {
+        List<Document> documents = event.getDocuments();
+        // Process the retrieved documents...
+    }
+}
+```
+
+{{< /tab >}}
+
+{{< /tabs >}}
+

Review Comment:
   Shall we also add instructions for building custom vector stores in Java?



##########
docs/content/docs/development/vector_stores.md:
##########
@@ -25,7 +25,7 @@ under the License.
 # Vector Stores
 
 {{< hint info >}}
-Vector stores are currently supported in the Python API only. Java API support is planned for future releases.
+Vector stores are supported in both Python and Java APIs.

Review Comment:
   Maybe we can just remove this hint?



##########
docs/content/docs/development/vector_stores.md:
##########
@@ -265,6 +285,105 @@ def chroma_store() -> ResourceDescriptor:
     )
 ```
 
+### Elasticsearch
+
+[Elasticsearch](https://www.elastic.co/elasticsearch/) is a distributed, RESTful search and analytics engine that supports vector search through dense vector fields and K-Nearest Neighbors (KNN).
+
+{{< hint info >}}
+Elasticsearch is currently supported in the Java API only.
+{{< /hint >}}
+
+#### Prerequisites
+
+1. An Elasticsearch cluster (version 8.0 or later for KNN support).
+2. An index with a `dense_vector` field.
+
+#### ElasticsearchVectorStore Parameters
+
+{{< tabs "ElasticsearchVectorStore Parameters" >}}
+
+{{< tab "Java" >}}
+
+| Parameter | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `embedding_model` | str | Required | Reference to embedding model resource name |
+| `index` | str | Required | Target Elasticsearch index name |
+| `vector_field` | str | Required | Name of the dense vector field used for KNN |
+| `dims` | int | `768` | Vector dimensionality |
+| `k` | int | None | Number of nearest neighbors to return; can be overridden per query |
+| `num_candidates` | int | None | Candidate set size for ANN search; can be overridden per query |
+| `filter_query` | str | None | Raw JSON Elasticsearch filter query (DSL) applied as a post-filter |
+| `host` | str | `"http://localhost:9200"` | Elasticsearch endpoint |
+| `hosts` | str | None | Comma-separated list of Elasticsearch endpoints |
+| `username` | str | None | Username for basic authentication |
+| `password` | str | None | Password for basic authentication |
+| `api_key_base64` | str | None | Base64-encoded API key for authentication |
+| `api_key_id` | str | None | API key ID for authentication |
+| `api_key_secret` | str | None | API key secret for authentication |
+
+{{< /tab >}}
+
+{{< /tabs >}}
+
+#### Usage Example
+
+{{< tabs "Elasticsearch Usage Example" >}}
+
+{{< tab "Java" >}}
+
+Here's how to define an Elasticsearch vector store in your Java agent:
+
+```java
+public class MyAgent extends Agent {
+
+    @EmbeddingModelConnection
+    public static ResourceDescriptor embeddingConnection() {
+        return ResourceDescriptor.Builder.newBuilder(OpenAIEmbeddingModelConnection.class.getName())
+                .addInitialArgument("api_key", "your-api-key-here")
+                .build();
+    }
+
+    @EmbeddingModelSetup
+    public static ResourceDescriptor embeddingModel() {
+        return ResourceDescriptor.Builder.newBuilder(OpenAIEmbeddingModelSetup.class.getName())
+                .addInitialArgument("connection", "embeddingConnection")
+                .addInitialArgument("model", "text-embedding-3-small")
+                .build();
+    }
+
+    @VectorStore
+    public static ResourceDescriptor vectorStore() {
+        return ResourceDescriptor.Builder.newBuilder(ElasticsearchVectorStore.class.getName())
+                .addInitialArgument("embedding_model", "embeddingModel")
+                .addInitialArgument("host", "http://localhost:9200")
+                .addInitialArgument("index", "my_documents")
+                .addInitialArgument("vector_field", "content_vector")
+                .addInitialArgument("dims", 1536)
+                // Optional authentication
+                // .addInitialArgument("username", "elastic")
+                // .addInitialArgument("password", "secret")
+                .build();
+    }
+
+    @Action(listenEvents = InputEvent.class)
+    public static void searchDocuments(InputEvent event, RunnerContext ctx) {
+        String query = (String) event.getInput();
+        // Request context retrieval via the vector store
+        ctx.sendEvent(new ContextRetrievalRequestEvent(query, "vectorStore"));
+    }
+
+    @Action(listenEvents = ContextRetrievalResponseEvent.class)
+    public static void onSearchResponse(ContextRetrievalResponseEvent event, RunnerContext ctx) {
+        List<Document> documents = event.getDocuments();
+        // Process the retrieved documents...
+    }
+}

Review Comment:
   Part of this example (i.e., how to search in the vector store with built-in events) is not specific to ES. We may consider modifying the `Getting Started` section and describe how to do the steps in Java. Then in the `Elasticsearch` section example, we can keep only the ES specific contents (i.e. creating the resource descriptor). Imagine that in future we may support many vector stores, so it might be better to avoid repeated contents as much as possible. WDYT?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] doc : add ElasticsearchVectorStore doc [flink-agents]

Posted by "twosom (via GitHub)" <gi...@apache.org>.
twosom commented on PR #397:
URL: https://github.com/apache/flink-agents/pull/397#issuecomment-3708780888

   @xintongsong
   
   Thank you for the review.
   Sorry for the late response. I had some personal matters to attend to.
   
   I will update the PR today.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] doc : add ElasticsearchVectorStore doc [flink-agents]

Posted by "twosom (via GitHub)" <gi...@apache.org>.
twosom commented on code in PR #397:
URL: https://github.com/apache/flink-agents/pull/397#discussion_r2678378394


##########
docs/content/docs/development/vector_stores.md:
##########
@@ -67,26 +83,64 @@ query = VectorStoreQuery(
 )
 ```
 
-### Query Results
+{{< /tab >}}
 
-When you execute a query, you receive a `VectorStoreQueryResult` object that contains the search results:
+{{< tab "Java" >}}
 
-```python
-# Execute the query
-result = vector_store.query(query)
+```java
+// Create a semantic search query
+VectorStoreQuery query = new VectorStoreQuery(
+        VectorStoreQueryMode.SEMANTIC,
+        "What is Apache Flink Agents?",
+        3
+);

Review Comment:
   @xintongsong 
   That makes sense! I'll update the documentation to use the constructor without the mode argument, keeping it consistent with the Python implementation. Will also comment out the unsupported modes in the Java enum.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] doc : add ElasticsearchVectorStore doc [flink-agents]

Posted by "twosom (via GitHub)" <gi...@apache.org>.
twosom commented on code in PR #397:
URL: https://github.com/apache/flink-agents/pull/397#discussion_r2672634906


##########
docs/content/docs/development/vector_stores.md:
##########
@@ -67,26 +83,64 @@ query = VectorStoreQuery(
 )
 ```
 
-### Query Results
+{{< /tab >}}
 
-When you execute a query, you receive a `VectorStoreQueryResult` object that contains the search results:
+{{< tab "Java" >}}
 
-```python
-# Execute the query
-result = vector_store.query(query)
+```java
+// Create a semantic search query
+VectorStoreQuery query = new VectorStoreQuery(
+        VectorStoreQueryMode.SEMANTIC,
+        "What is Apache Flink Agents?",
+        3
+);

Review Comment:
   @xintongsong 
   
   Thank you for the review!
   
   I apologize - I realize now that this constructor doesn't actually exist in the current implementation. I will add it to support this cleaner API.
   
   I'll also apply your suggested inline comments.
   
   Will update shortly.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org