You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@answer.apache.org by "hgaol (via GitHub)" <gi...@apache.org> on 2026/05/11 15:09:33 UTC
[PR] feat: add 6 vector-search backend plugins [answer-plugins]
hgaol opened a new pull request, #320:
URL: https://github.com/apache/answer-plugins/pull/320
Close #319
1. Added 6 new plugins: pgvector, qdrant, milvus, weaviate, elasticsearch, chromadb. All are under answer-plugins/vector-search-*/
2. Validated all plugins with local docker containers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@answer.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: add 6 vector-search backend plugins [answer-plugins]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #320:
URL: https://github.com/apache/answer-plugins/pull/320#discussion_r3533491882
##########
vector-search-weaviate/sync.go:
##########
@@ -0,0 +1,105 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package weaviate_vector_search
+
+import (
+ "context"
+
+ "github.com/apache/answer/plugin"
+ "github.com/segmentfault/pacman/log"
+)
+
+// sync triggers a full bulk sync of all Q&A content into Weaviate.
+// It runs in a background goroutine and is guarded by the syncing flag.
+func (e *VectorSearchEngine) sync() {
+ if e.syncer == nil {
+ log.Warn("weaviate: syncer not registered, skip sync")
+ return
+ }
+ if e.syncing {
+ log.Warn("weaviate: sync already running, skip")
+ return
+ }
+
+ go func() {
+ e.lock.Lock()
+ defer e.lock.Unlock()
+ if e.syncing {
+ return
+ }
+ e.syncing = true
+ defer func() { e.syncing = false }()
+
Review Comment:
`e.syncing` is read/written without synchronization outside the mutex, which can cause a data race and can still allow multiple sync goroutines to start concurrently. Consider guarding the flag with the mutex and setting it before launching the goroutine.
##########
vector-search-pgvector/pgvector.go:
##########
@@ -0,0 +1,398 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package pgvector_search
+
+import (
+ "context"
+ "database/sql"
+ "embed"
+ "encoding/json"
+ "fmt"
+ "strconv"
+ "sync"
+
+ "github.com/apache/answer-plugins/util"
+ "github.com/apache/answer-plugins/vector-search-pgvector/i18n"
+ "github.com/apache/answer/plugin"
+ _ "github.com/lib/pq"
+ pgv "github.com/pgvector/pgvector-go"
+ "github.com/segmentfault/pacman/log"
+)
+
+//go:embed info.yaml
+var Info embed.FS
+
+const tableName = "answer_vector_embeddings"
+
+// VectorSearchEngine implements plugin.VectorSearch using PostgreSQL + pgvector.
+type VectorSearchEngine struct {
+ Config *VectorSearchConfig
+ db *sql.DB
+ syncer plugin.VectorSearchSyncer
+ syncing bool
+ lock sync.Mutex
+ embeddingDimensions int // auto-detected from embedding model
+}
+
+// VectorSearchConfig holds all plugin configuration.
+type VectorSearchConfig struct {
+ DSN string `json:"dsn"`
+ APIHost string `json:"api_host"`
+ APIKey string `json:"api_key"`
+ EmbeddingModel string `json:"embedding_model"`
+ EmbeddingLevel string `json:"embedding_level"`
+ SimilarityThreshold float64 `json:"similarity_threshold"`
+}
+
+func init() {
+ plugin.Register(&VectorSearchEngine{
+ Config: &VectorSearchConfig{},
+ lock: sync.Mutex{},
+ })
+}
+
+func (e *VectorSearchEngine) Info() plugin.Info {
+ info := &util.Info{}
+ info.GetInfo(Info)
+
+ return plugin.Info{
+ Name: plugin.MakeTranslator(i18n.InfoName),
+ SlugName: info.SlugName,
+ Description: plugin.MakeTranslator(i18n.InfoDescription),
+ Author: info.Author,
+ Version: info.Version,
+ Link: info.Link,
+ }
+}
+
+// Description returns metadata about this vector search engine.
+func (e *VectorSearchEngine) Description() plugin.VectorSearchDesc {
+ return plugin.VectorSearchDesc{
+ Icon: "",
+ Link: "https://github.com/pgvector/pgvector",
+ }
+}
+
+// RegisterSyncer stores the syncer and triggers a full sync.
+func (e *VectorSearchEngine) RegisterSyncer(ctx context.Context, syncer plugin.VectorSearchSyncer) {
+ log.Debugf("pgvector: RegisterSyncer called, db=%v, embeddingLevel=%s", e.db != nil, e.Config.EmbeddingLevel)
+ e.syncer = syncer
+ if e.db != nil {
+ e.sync()
+ }
+}
+
+// SearchSimilar performs a cosine similarity search using pgvector.
+func (e *VectorSearchEngine) SearchSimilar(ctx context.Context, query string, topK int) ([]plugin.VectorSearchResult, error) {
+ if e.db == nil {
+ return nil, fmt.Errorf("pgvector: database not initialized")
+ }
+ if topK <= 0 {
+ topK = 10
+ }
+
+ log.Debugf("pgvector: SearchSimilar query=%q topK=%d model=%s", query, topK, e.Config.EmbeddingModel)
+
+ embedding, err := plugin.GenerateEmbedding(ctx, e.Config.APIHost, e.Config.APIKey, e.Config.EmbeddingModel, query)
+ if err != nil {
+ return nil, fmt.Errorf("generate query embedding: %w", err)
+ }
+
+ log.Debugf("pgvector: search embedding generated, dimensions=%d", len(embedding))
+
+ vec := pgv.NewVector(embedding)
+
+ // Use cosine distance operator (<=>). Score = 1 - distance.
+ sqlQuery := fmt.Sprintf(
+ `SELECT object_id, object_type, metadata, 1 - (embedding <=> $1) AS score
+ FROM %s
+ ORDER BY embedding <=> $1
+ LIMIT $2`, tableName)
+
+ rows, err := e.db.QueryContext(ctx, sqlQuery, vec, topK)
+ if err != nil {
+ return nil, fmt.Errorf("pgvector search failed: %w", err)
+ }
+ defer rows.Close()
+
+ results := make([]plugin.VectorSearchResult, 0, topK)
+ for rows.Next() {
+ var r plugin.VectorSearchResult
+ if err := rows.Scan(&r.ObjectID, &r.ObjectType, &r.Metadata, &r.Score); err != nil {
+ log.Warnf("pgvector: scan row: %v", err)
+ continue
+ }
+ if e.Config.SimilarityThreshold > 0 && r.Score < e.Config.SimilarityThreshold {
+ log.Debugf("pgvector: skipping result %s score=%.4f below threshold=%.4f", r.ObjectID, r.Score, e.Config.SimilarityThreshold)
+ continue
+ }
+ results = append(results, r)
+ }
+ log.Debugf("pgvector: SearchSimilar returning %d results", len(results))
+ return results, rows.Err()
+}
+
+// UpdateContent upserts a single document into the pgvector table.
+func (e *VectorSearchEngine) UpdateContent(ctx context.Context, content *plugin.VectorSearchContent) error {
+ if e.db == nil {
+ return fmt.Errorf("pgvector: database not initialized")
+ }
+
+ log.Debugf("pgvector: UpdateContent objectID=%s objectType=%s titleLen=%d contentLen=%d",
+ content.ObjectID, content.ObjectType, len(content.Title), len(content.Content))
+
+ embedding, err := plugin.GenerateEmbedding(ctx, e.Config.APIHost, e.Config.APIKey, e.Config.EmbeddingModel, content.Content)
+ if err != nil {
+ return fmt.Errorf("generate embedding for %s: %w", content.ObjectID, err)
+ }
+
+ log.Debugf("pgvector: embedding generated for %s, dimensions=%d", content.ObjectID, len(embedding))
+
+ vec := pgv.NewVector(embedding)
+
+ query := fmt.Sprintf(
+ `INSERT INTO %s (object_id, object_type, title, content, metadata, embedding)
+ VALUES ($1, $2, $3, $4, $5, $6)
+ ON CONFLICT (object_id) DO UPDATE SET
+ object_type = EXCLUDED.object_type,
+ title = EXCLUDED.title,
+ content = EXCLUDED.content,
+ metadata = EXCLUDED.metadata,
+ embedding = EXCLUDED.embedding`, tableName)
+
+ _, err = e.db.ExecContext(ctx, query,
+ content.ObjectID, content.ObjectType, content.Title, content.Content, content.Metadata, vec)
+ if err != nil {
+ return fmt.Errorf("upsert document %s: %w", content.ObjectID, err)
+ }
+ log.Debugf("pgvector: upserted document %s successfully", content.ObjectID)
+ return nil
+}
+
+// DeleteContent removes a document by object ID.
+func (e *VectorSearchEngine) DeleteContent(ctx context.Context, objectID string) error {
+ if e.db == nil {
+ return fmt.Errorf("pgvector: database not initialized")
+ }
+ log.Debugf("pgvector: DeleteContent objectID=%s", objectID)
+ _, err := e.db.ExecContext(ctx, fmt.Sprintf(`DELETE FROM %s WHERE object_id = $1`, tableName), objectID)
+ if err != nil {
+ return fmt.Errorf("delete document %s: %w", objectID, err)
+ }
+ log.Debugf("pgvector: deleted document %s", objectID)
+ return nil
+}
+
+// ConfigFields returns the plugin configuration form fields.
+func (e *VectorSearchEngine) ConfigFields() []plugin.ConfigField {
+ return []plugin.ConfigField{
+ {
+ Name: "dsn",
+ Type: plugin.ConfigTypeInput,
+ Title: plugin.MakeTranslator(i18n.ConfigDSNTitle),
+ Description: plugin.MakeTranslator(i18n.ConfigDSNDescription),
+ Required: true,
+ UIOptions: plugin.ConfigFieldUIOptions{
+ InputType: plugin.InputTypeText,
+ },
+ Value: e.Config.DSN,
+ },
+ {
+ Name: "api_host",
+ Type: plugin.ConfigTypeInput,
+ Title: plugin.MakeTranslator(i18n.ConfigAPIHostTitle),
+ Description: plugin.MakeTranslator(i18n.ConfigAPIHostDescription),
+ Required: true,
+ UIOptions: plugin.ConfigFieldUIOptions{
+ InputType: plugin.InputTypeText,
+ },
+ Value: e.Config.APIHost,
+ },
+ {
+ Name: "api_key",
+ Type: plugin.ConfigTypeInput,
+ Title: plugin.MakeTranslator(i18n.ConfigAPIKeyTitle),
+ Description: plugin.MakeTranslator(i18n.ConfigAPIKeyDescription),
+ Required: true,
+ UIOptions: plugin.ConfigFieldUIOptions{
+ InputType: plugin.InputTypePassword,
+ },
+ Value: e.Config.APIKey,
+ },
+ {
+ Name: "embedding_model",
+ Type: plugin.ConfigTypeInput,
+ Title: plugin.MakeTranslator(i18n.ConfigEmbeddingModelTitle),
+ Description: plugin.MakeTranslator(i18n.ConfigEmbeddingModelDescription),
+ Required: true,
+ UIOptions: plugin.ConfigFieldUIOptions{
+ InputType: plugin.InputTypeText,
+ },
+ Value: e.Config.EmbeddingModel,
+ },
+ {
+ Name: "embedding_level",
+ Type: plugin.ConfigTypeSelect,
+ Title: plugin.MakeTranslator(i18n.ConfigEmbeddingLevelTitle),
+ Description: plugin.MakeTranslator(i18n.ConfigEmbeddingLevelDescription),
+ Required: true,
+ Options: []plugin.ConfigFieldOption{
+ {Label: plugin.MakeTranslator(i18n.ConfigEmbeddingLevelOptionQuestion), Value: "question"},
+ {Label: plugin.MakeTranslator(i18n.ConfigEmbeddingLevelOptionAnswer), Value: "answer"},
+ },
+ Value: e.Config.EmbeddingLevel,
+ },
+ {
+ Name: "similarity_threshold",
+ Type: plugin.ConfigTypeInput,
+ Title: plugin.MakeTranslator(i18n.ConfigSimilarityThresholdTitle),
+ Description: plugin.MakeTranslator(i18n.ConfigSimilarityThresholdDescription),
+ Required: false,
+ UIOptions: plugin.ConfigFieldUIOptions{
+ InputType: plugin.InputTypeText,
+ },
+ Value: fmt.Sprintf("%.2f", e.Config.SimilarityThreshold),
+ },
+ }
+}
+
+// ConfigReceiver applies configuration from the admin UI.
+func (e *VectorSearchEngine) ConfigReceiver(config []byte) error {
+ log.Debugf("pgvector: ConfigReceiver called, raw config=%s", string(config))
Review Comment:
This debug log prints the entire raw config payload, which can include secrets (e.g., embedding API key) and credentials embedded in the DSN. Please avoid logging raw config values.
##########
vector-search-qdrant/sync.go:
##########
@@ -0,0 +1,102 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package qdrant_vector_search
+
+import (
+ "context"
+
+ "github.com/apache/answer/plugin"
+ "github.com/segmentfault/pacman/log"
+)
+
+func (e *VectorSearchEngine) sync() {
+ if e.syncer == nil {
+ log.Warn("qdrant: syncer not registered, skip sync")
+ return
+ }
+ if e.syncing {
+ log.Warn("qdrant: sync already running, skip")
+ return
+ }
+
+ go func() {
+ e.lock.Lock()
+ defer e.lock.Unlock()
+ if e.syncing {
+ return
+ }
+ e.syncing = true
+ defer func() { e.syncing = false }()
+
Review Comment:
`e.syncing` is checked without holding the mutex, which can lead to a data race and multiple sync goroutines starting concurrently. Guard the flag with the mutex and set it before launching the goroutine.
##########
vector-search-pgvector/sync.go:
##########
@@ -0,0 +1,105 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package pgvector_search
+
+import (
+ "context"
+
+ "github.com/apache/answer/plugin"
+ "github.com/segmentfault/pacman/log"
+)
+
+// sync triggers a full bulk sync of all Q&A content into the pgvector table.
+// It runs in a background goroutine and is guarded by the syncing flag.
+func (e *VectorSearchEngine) sync() {
+ if e.syncer == nil {
+ log.Warn("pgvector: syncer not registered, skip sync")
+ return
+ }
+ if e.syncing {
+ log.Warn("pgvector: sync already running, skip")
+ return
+ }
+
+ go func() {
+ e.lock.Lock()
+ defer e.lock.Unlock()
+ if e.syncing {
+ return
+ }
+ e.syncing = true
+ defer func() { e.syncing = false }()
+
Review Comment:
`e.syncing` is checked without holding the mutex, which can lead to a data race and multiple sync goroutines starting concurrently. Guard the flag with the mutex and set it before launching the goroutine.
##########
vector-search-chromadb/sync.go:
##########
@@ -0,0 +1,102 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package chromadb_vector_search
+
+import (
+ "context"
+
+ "github.com/apache/answer/plugin"
+ "github.com/segmentfault/pacman/log"
+)
+
+func (e *VectorSearchEngine) sync() {
+ if e.syncer == nil {
+ log.Warn("chromadb: syncer not registered, skip sync")
+ return
+ }
+ if e.syncing {
+ log.Warn("chromadb: sync already running, skip")
+ return
+ }
+
+ go func() {
+ e.lock.Lock()
+ defer e.lock.Unlock()
+ if e.syncing {
+ return
+ }
+ e.syncing = true
+ defer func() { e.syncing = false }()
Review Comment:
`e.syncing` is checked without holding the mutex, which can lead to a data race and multiple sync goroutines starting concurrently. Guard the flag with the mutex and set it before launching the goroutine.
##########
vector-search-elasticsearch/sync.go:
##########
@@ -0,0 +1,99 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package es_vector_search
+
+import (
+ "context"
+
+ "github.com/apache/answer/plugin"
+ "github.com/segmentfault/pacman/log"
+)
+
+// sync triggers a full bulk sync of all Q&A content into the vector store.
+// It runs in a background goroutine and is guarded by the syncing flag.
+func (e *VectorSearchEngine) sync() {
+ if e.syncer == nil {
+ log.Warn("es-vector: syncer not registered, skip sync")
+ return
+ }
+ if e.syncing {
+ log.Warn("es-vector: sync already running, skip")
+ return
+ }
+
+ go func() {
+ e.lock.Lock()
+ defer e.lock.Unlock()
+ if e.syncing {
+ return
+ }
+ e.syncing = true
+ defer func() { e.syncing = false }()
Review Comment:
`e.syncing` is checked without holding the mutex, which can lead to a data race and multiple sync goroutines starting concurrently. Guard the flag with the mutex and set it before launching the goroutine.
##########
vector-search-milvus/sync.go:
##########
@@ -0,0 +1,102 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package milvus_vector_search
+
+import (
+ "context"
+
+ "github.com/apache/answer/plugin"
+ "github.com/segmentfault/pacman/log"
+)
+
+func (e *VectorSearchEngine) sync() {
+ if e.syncer == nil {
+ log.Warn("milvus: syncer not registered, skip sync")
+ return
+ }
+ if e.syncing {
+ log.Warn("milvus: sync already running, skip")
+ return
+ }
+
+ go func() {
+ e.lock.Lock()
+ defer e.lock.Unlock()
+ if e.syncing {
+ return
+ }
+ e.syncing = true
+ defer func() { e.syncing = false }()
Review Comment:
`e.syncing` is checked without holding the mutex, which can lead to a data race and multiple sync goroutines starting concurrently. Guard the flag with the mutex and set it before launching the goroutine.
##########
vector-search-pgvector/pgvector.go:
##########
@@ -0,0 +1,398 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package pgvector_search
+
+import (
+ "context"
+ "database/sql"
+ "embed"
+ "encoding/json"
+ "fmt"
+ "strconv"
+ "sync"
+
+ "github.com/apache/answer-plugins/util"
+ "github.com/apache/answer-plugins/vector-search-pgvector/i18n"
+ "github.com/apache/answer/plugin"
+ _ "github.com/lib/pq"
+ pgv "github.com/pgvector/pgvector-go"
+ "github.com/segmentfault/pacman/log"
+)
+
+//go:embed info.yaml
+var Info embed.FS
+
+const tableName = "answer_vector_embeddings"
+
+// VectorSearchEngine implements plugin.VectorSearch using PostgreSQL + pgvector.
+type VectorSearchEngine struct {
+ Config *VectorSearchConfig
+ db *sql.DB
+ syncer plugin.VectorSearchSyncer
+ syncing bool
+ lock sync.Mutex
+ embeddingDimensions int // auto-detected from embedding model
+}
+
+// VectorSearchConfig holds all plugin configuration.
+type VectorSearchConfig struct {
+ DSN string `json:"dsn"`
+ APIHost string `json:"api_host"`
+ APIKey string `json:"api_key"`
+ EmbeddingModel string `json:"embedding_model"`
+ EmbeddingLevel string `json:"embedding_level"`
+ SimilarityThreshold float64 `json:"similarity_threshold"`
+}
+
+func init() {
+ plugin.Register(&VectorSearchEngine{
+ Config: &VectorSearchConfig{},
+ lock: sync.Mutex{},
+ })
+}
+
+func (e *VectorSearchEngine) Info() plugin.Info {
+ info := &util.Info{}
+ info.GetInfo(Info)
+
+ return plugin.Info{
+ Name: plugin.MakeTranslator(i18n.InfoName),
+ SlugName: info.SlugName,
+ Description: plugin.MakeTranslator(i18n.InfoDescription),
+ Author: info.Author,
+ Version: info.Version,
+ Link: info.Link,
+ }
+}
+
+// Description returns metadata about this vector search engine.
+func (e *VectorSearchEngine) Description() plugin.VectorSearchDesc {
+ return plugin.VectorSearchDesc{
+ Icon: "",
+ Link: "https://github.com/pgvector/pgvector",
+ }
+}
+
+// RegisterSyncer stores the syncer and triggers a full sync.
+func (e *VectorSearchEngine) RegisterSyncer(ctx context.Context, syncer plugin.VectorSearchSyncer) {
+ log.Debugf("pgvector: RegisterSyncer called, db=%v, embeddingLevel=%s", e.db != nil, e.Config.EmbeddingLevel)
+ e.syncer = syncer
+ if e.db != nil {
+ e.sync()
+ }
+}
+
+// SearchSimilar performs a cosine similarity search using pgvector.
+func (e *VectorSearchEngine) SearchSimilar(ctx context.Context, query string, topK int) ([]plugin.VectorSearchResult, error) {
+ if e.db == nil {
+ return nil, fmt.Errorf("pgvector: database not initialized")
+ }
+ if topK <= 0 {
+ topK = 10
+ }
+
+ log.Debugf("pgvector: SearchSimilar query=%q topK=%d model=%s", query, topK, e.Config.EmbeddingModel)
+
+ embedding, err := plugin.GenerateEmbedding(ctx, e.Config.APIHost, e.Config.APIKey, e.Config.EmbeddingModel, query)
+ if err != nil {
+ return nil, fmt.Errorf("generate query embedding: %w", err)
+ }
+
+ log.Debugf("pgvector: search embedding generated, dimensions=%d", len(embedding))
+
+ vec := pgv.NewVector(embedding)
+
+ // Use cosine distance operator (<=>). Score = 1 - distance.
+ sqlQuery := fmt.Sprintf(
+ `SELECT object_id, object_type, metadata, 1 - (embedding <=> $1) AS score
+ FROM %s
+ ORDER BY embedding <=> $1
+ LIMIT $2`, tableName)
+
+ rows, err := e.db.QueryContext(ctx, sqlQuery, vec, topK)
+ if err != nil {
+ return nil, fmt.Errorf("pgvector search failed: %w", err)
+ }
+ defer rows.Close()
+
+ results := make([]plugin.VectorSearchResult, 0, topK)
+ for rows.Next() {
+ var r plugin.VectorSearchResult
+ if err := rows.Scan(&r.ObjectID, &r.ObjectType, &r.Metadata, &r.Score); err != nil {
+ log.Warnf("pgvector: scan row: %v", err)
+ continue
+ }
+ if e.Config.SimilarityThreshold > 0 && r.Score < e.Config.SimilarityThreshold {
+ log.Debugf("pgvector: skipping result %s score=%.4f below threshold=%.4f", r.ObjectID, r.Score, e.Config.SimilarityThreshold)
+ continue
+ }
+ results = append(results, r)
+ }
+ log.Debugf("pgvector: SearchSimilar returning %d results", len(results))
+ return results, rows.Err()
+}
+
+// UpdateContent upserts a single document into the pgvector table.
+func (e *VectorSearchEngine) UpdateContent(ctx context.Context, content *plugin.VectorSearchContent) error {
+ if e.db == nil {
+ return fmt.Errorf("pgvector: database not initialized")
+ }
+
+ log.Debugf("pgvector: UpdateContent objectID=%s objectType=%s titleLen=%d contentLen=%d",
+ content.ObjectID, content.ObjectType, len(content.Title), len(content.Content))
+
+ embedding, err := plugin.GenerateEmbedding(ctx, e.Config.APIHost, e.Config.APIKey, e.Config.EmbeddingModel, content.Content)
+ if err != nil {
+ return fmt.Errorf("generate embedding for %s: %w", content.ObjectID, err)
+ }
+
+ log.Debugf("pgvector: embedding generated for %s, dimensions=%d", content.ObjectID, len(embedding))
+
+ vec := pgv.NewVector(embedding)
+
+ query := fmt.Sprintf(
+ `INSERT INTO %s (object_id, object_type, title, content, metadata, embedding)
+ VALUES ($1, $2, $3, $4, $5, $6)
+ ON CONFLICT (object_id) DO UPDATE SET
+ object_type = EXCLUDED.object_type,
+ title = EXCLUDED.title,
+ content = EXCLUDED.content,
+ metadata = EXCLUDED.metadata,
+ embedding = EXCLUDED.embedding`, tableName)
+
+ _, err = e.db.ExecContext(ctx, query,
+ content.ObjectID, content.ObjectType, content.Title, content.Content, content.Metadata, vec)
+ if err != nil {
+ return fmt.Errorf("upsert document %s: %w", content.ObjectID, err)
+ }
+ log.Debugf("pgvector: upserted document %s successfully", content.ObjectID)
+ return nil
+}
+
+// DeleteContent removes a document by object ID.
+func (e *VectorSearchEngine) DeleteContent(ctx context.Context, objectID string) error {
+ if e.db == nil {
+ return fmt.Errorf("pgvector: database not initialized")
+ }
+ log.Debugf("pgvector: DeleteContent objectID=%s", objectID)
+ _, err := e.db.ExecContext(ctx, fmt.Sprintf(`DELETE FROM %s WHERE object_id = $1`, tableName), objectID)
+ if err != nil {
+ return fmt.Errorf("delete document %s: %w", objectID, err)
+ }
+ log.Debugf("pgvector: deleted document %s", objectID)
+ return nil
+}
+
+// ConfigFields returns the plugin configuration form fields.
+func (e *VectorSearchEngine) ConfigFields() []plugin.ConfigField {
+ return []plugin.ConfigField{
+ {
+ Name: "dsn",
+ Type: plugin.ConfigTypeInput,
+ Title: plugin.MakeTranslator(i18n.ConfigDSNTitle),
+ Description: plugin.MakeTranslator(i18n.ConfigDSNDescription),
+ Required: true,
+ UIOptions: plugin.ConfigFieldUIOptions{
+ InputType: plugin.InputTypeText,
+ },
+ Value: e.Config.DSN,
+ },
+ {
+ Name: "api_host",
+ Type: plugin.ConfigTypeInput,
+ Title: plugin.MakeTranslator(i18n.ConfigAPIHostTitle),
+ Description: plugin.MakeTranslator(i18n.ConfigAPIHostDescription),
+ Required: true,
+ UIOptions: plugin.ConfigFieldUIOptions{
+ InputType: plugin.InputTypeText,
+ },
+ Value: e.Config.APIHost,
+ },
+ {
+ Name: "api_key",
+ Type: plugin.ConfigTypeInput,
+ Title: plugin.MakeTranslator(i18n.ConfigAPIKeyTitle),
+ Description: plugin.MakeTranslator(i18n.ConfigAPIKeyDescription),
+ Required: true,
+ UIOptions: plugin.ConfigFieldUIOptions{
+ InputType: plugin.InputTypePassword,
+ },
+ Value: e.Config.APIKey,
+ },
+ {
+ Name: "embedding_model",
+ Type: plugin.ConfigTypeInput,
+ Title: plugin.MakeTranslator(i18n.ConfigEmbeddingModelTitle),
+ Description: plugin.MakeTranslator(i18n.ConfigEmbeddingModelDescription),
+ Required: true,
+ UIOptions: plugin.ConfigFieldUIOptions{
+ InputType: plugin.InputTypeText,
+ },
+ Value: e.Config.EmbeddingModel,
+ },
+ {
+ Name: "embedding_level",
+ Type: plugin.ConfigTypeSelect,
+ Title: plugin.MakeTranslator(i18n.ConfigEmbeddingLevelTitle),
+ Description: plugin.MakeTranslator(i18n.ConfigEmbeddingLevelDescription),
+ Required: true,
+ Options: []plugin.ConfigFieldOption{
+ {Label: plugin.MakeTranslator(i18n.ConfigEmbeddingLevelOptionQuestion), Value: "question"},
+ {Label: plugin.MakeTranslator(i18n.ConfigEmbeddingLevelOptionAnswer), Value: "answer"},
+ },
+ Value: e.Config.EmbeddingLevel,
+ },
+ {
+ Name: "similarity_threshold",
+ Type: plugin.ConfigTypeInput,
+ Title: plugin.MakeTranslator(i18n.ConfigSimilarityThresholdTitle),
+ Description: plugin.MakeTranslator(i18n.ConfigSimilarityThresholdDescription),
+ Required: false,
+ UIOptions: plugin.ConfigFieldUIOptions{
+ InputType: plugin.InputTypeText,
+ },
+ Value: fmt.Sprintf("%.2f", e.Config.SimilarityThreshold),
+ },
+ }
+}
+
+// ConfigReceiver applies configuration from the admin UI.
+func (e *VectorSearchEngine) ConfigReceiver(config []byte) error {
+ log.Debugf("pgvector: ConfigReceiver called, raw config=%s", string(config))
+
+ // Pre-process: convert string numbers to actual numbers before unmarshalling,
+ // because the admin UI sends all form values as strings.
+ var raw map[string]interface{}
+ if err := json.Unmarshal(config, &raw); err != nil {
+ return fmt.Errorf("unmarshal config: %w", err)
+ }
+ if threshStr, ok := raw["similarity_threshold"].(string); ok {
+ if thresh, err := strconv.ParseFloat(threshStr, 64); err == nil {
+ raw["similarity_threshold"] = thresh
+ }
+ }
+ fixed, _ := json.Marshal(raw)
+
+ conf := &VectorSearchConfig{}
+ if err := json.Unmarshal(fixed, conf); err != nil {
+ return fmt.Errorf("unmarshal config: %w", err)
+ }
+
+ e.Config = conf
+
+ log.Debugf("pgvector: config parsed: dsn=%s model=%s level=%s threshold=%.2f",
+ conf.DSN, conf.EmbeddingModel, conf.EmbeddingLevel, conf.SimilarityThreshold)
Review Comment:
This debug log prints the full PostgreSQL DSN, which may contain usernames/passwords. Logging connection strings is a credential leak; please log only whether a DSN is set (or a redacted form).
##########
vector-search-elasticsearch/README.md:
##########
@@ -0,0 +1,43 @@
+# Apache Answer Plugin: Elasticsearch Vector Search
+
+This plugin enables semantic/vector search in [Apache Answer](https://github.com/apache/answer) using [Elasticsearch](https://www.elastic.co/elasticsearch/) with dense vector fields.
+
+## Prerequisites
+
+- Elasticsearch 8.0+ (with dense vector and kNN search support)
+- An OpenAI-compatible embedding API (e.g., OpenAI, Azure OpenAI, or any compatible provider)
+
+## Installation
+
+Build Apache Answer with this plugin:
+
+```bash
+./answer build --with github.com/apache/answer-plugins/vector-search-elasticsearch
+```
+
+## Configuration
+
+After enabling the plugin in the Admin UI (**Admin > Plugins > Vector Search**), configure the following fields:
+
+| Field | Description | Example |
+|---|---|---|
+| **Endpoints** | Comma-separated Elasticsearch URLs | `http://localhost:9200` |
+| **Username** | Elasticsearch username (optional) | `elastic` |
+| **Password** | Elasticsearch password (optional) | `changeme` |
+| **Embedding API Host** | OpenAI-compatible API base URL | `https://api.openai.com` |
+| **Embedding API Key** | API key for the embedding service | `sk-...` |
+| **Embedding Model** | Model name for generating embeddings | `text-embedding-3-small` |
+| **Embedding Level** | `question` embeds question + all answers + comments together; `answer` embeds each answer separately | `question` |
+| **Similarity Threshold** | Minimum cosine similarity score (0-1). Default `0` means no filtering | `0.5` |
+
+## How It Works
+
+- **Embedding dimensions** are auto-detected from the configured model. No manual dimension configuration is needed.
+- On first configuration, the plugin creates an index `answer_vector` with a `dense_vector` field matching the detected dimensions and cosine similarity.
+- If the embedding model changes and produces different dimensions, the index is automatically deleted and recreated.
+- Uses Elasticsearch kNN search for vector similarity queries.
Review Comment:
The README claims the Elasticsearch index is automatically deleted and recreated when embedding dimensions change, but `ensureIndex` only creates the index if it doesn't exist and does not validate/recreate mappings/dims. Either implement dimension-change detection or update the README to match current behavior.
##########
vector-search-weaviate/weaviate.go:
##########
@@ -0,0 +1,551 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package weaviate_vector_search
+
+import (
+ "context"
+ "crypto/sha1"
+ "embed"
+ "encoding/json"
+ "fmt"
+ "strconv"
+ "strings"
+ "sync"
+
+ "github.com/apache/answer-plugins/util"
+ "github.com/apache/answer-plugins/vector-search-weaviate/i18n"
+ "github.com/apache/answer/plugin"
+ "github.com/segmentfault/pacman/log"
+ "github.com/weaviate/weaviate-go-client/v4/weaviate"
+ "github.com/weaviate/weaviate-go-client/v4/weaviate/auth"
+ "github.com/weaviate/weaviate-go-client/v4/weaviate/graphql"
+ "github.com/weaviate/weaviate/entities/models"
+)
+
+//go:embed info.yaml
+var Info embed.FS
+
+const (
+ className = "AnswerVector"
+)
+
+// VectorSearchEngine implements plugin.VectorSearch using Weaviate.
+type VectorSearchEngine struct {
+ Config *VectorSearchConfig
+ client *weaviate.Client
+ syncer plugin.VectorSearchSyncer
+ syncing bool
+ lock sync.Mutex
+}
+
+// VectorSearchConfig holds all plugin configuration.
+type VectorSearchConfig struct {
+ Endpoint string `json:"endpoint"`
+ APIKey string `json:"api_key"`
+ EmbeddingAPIHost string `json:"embedding_api_host"`
+ EmbeddingAPIKey string `json:"embedding_api_key"`
+ EmbeddingModel string `json:"embedding_model"`
+ EmbeddingLevel string `json:"embedding_level"`
+ SimilarityThreshold float64 `json:"similarity_threshold"`
+}
+
+func init() {
+ plugin.Register(&VectorSearchEngine{
+ Config: &VectorSearchConfig{},
+ lock: sync.Mutex{},
+ })
+}
+
+func (e *VectorSearchEngine) Info() plugin.Info {
+ info := &util.Info{}
+ info.GetInfo(Info)
+
+ return plugin.Info{
+ Name: plugin.MakeTranslator(i18n.InfoName),
+ SlugName: info.SlugName,
+ Description: plugin.MakeTranslator(i18n.InfoDescription),
+ Author: info.Author,
+ Version: info.Version,
+ Link: info.Link,
+ }
+}
+
+// Description returns metadata about this vector search engine.
+func (e *VectorSearchEngine) Description() plugin.VectorSearchDesc {
+ return plugin.VectorSearchDesc{
+ Icon: "",
+ Link: "https://weaviate.io/developers/weaviate",
+ }
+}
+
+// RegisterSyncer stores the syncer and triggers a full sync.
+func (e *VectorSearchEngine) RegisterSyncer(ctx context.Context, syncer plugin.VectorSearchSyncer) {
+ e.syncer = syncer
+ if e.client != nil {
+ e.sync()
+ }
+}
+
+// SearchSimilar performs a nearVector search in Weaviate.
+func (e *VectorSearchEngine) SearchSimilar(ctx context.Context, query string, topK int) ([]plugin.VectorSearchResult, error) {
+ if e.client == nil {
+ return nil, fmt.Errorf("weaviate client not initialized")
+ }
+ if topK <= 0 {
+ topK = 10
+ }
+
+ log.Debugf("weaviate: SearchSimilar query=%q topK=%d model=%s", query, topK, e.Config.EmbeddingModel)
+
+ // Generate embedding for the query text.
+ log.Debugf("weaviate: generating query embedding via %s model=%s", e.Config.EmbeddingAPIHost, e.Config.EmbeddingModel)
+ embedding, err := plugin.GenerateEmbedding(ctx, e.Config.EmbeddingAPIHost, e.Config.EmbeddingAPIKey, e.Config.EmbeddingModel, query)
+ if err != nil {
+ log.Errorf("weaviate: generate query embedding failed: %v", err)
+ return nil, fmt.Errorf("generate query embedding: %w", err)
+ }
+ log.Debugf("weaviate: query embedding generated, dimensions=%d", len(embedding))
+
+ embeddingFloats := make([]float32, len(embedding))
+ copy(embeddingFloats, embedding)
+
+ nearVector := e.client.GraphQL().NearVectorArgBuilder().WithVector(embeddingFloats)
+
+ fields := []graphql.Field{
+ {Name: "objectID"},
+ {Name: "objectType"},
+ {Name: "title"},
+ {Name: "metadata"},
+ {Name: "_additional", Fields: []graphql.Field{
+ {Name: "certainty"},
+ {Name: "id"},
+ }},
+ }
+
+ log.Debugf("weaviate: executing nearVector search on class=%s limit=%d", className, topK)
+ result, err := e.client.GraphQL().Get().
+ WithClassName(className).
+ WithFields(fields...).
+ WithNearVector(nearVector).
+ WithLimit(topK).
+ Do(ctx)
+ if err != nil {
+ log.Errorf("weaviate: nearVector search failed: %v", err)
+ return nil, fmt.Errorf("weaviate nearVector search failed: %w", err)
+ }
+
+ if result.Errors != nil {
+ msgs := make([]string, 0, len(result.Errors))
+ for _, e := range result.Errors {
+ msgs = append(msgs, e.Message)
+ }
+ log.Errorf("weaviate: GraphQL errors: %s", strings.Join(msgs, "; "))
+ return nil, fmt.Errorf("weaviate GraphQL errors: %s", strings.Join(msgs, "; "))
+ }
+
+ data, ok := result.Data["Get"].(map[string]interface{})
+ if !ok {
+ log.Warnf("weaviate: unexpected response format from GraphQL")
+ return nil, fmt.Errorf("unexpected response format")
+ }
+ objects, ok := data[className].([]interface{})
+ if !ok {
+ log.Debugf("weaviate: no objects returned from search")
+ return nil, nil
+ }
+
+ log.Debugf("weaviate: search returned %d raw results", len(objects))
+
+ results := make([]plugin.VectorSearchResult, 0, len(objects))
+ filtered := 0
+ for _, obj := range objects {
+ m, ok := obj.(map[string]interface{})
+ if !ok {
+ continue
+ }
+
+ objectID, _ := m["objectID"].(string)
+ objectType, _ := m["objectType"].(string)
+ metadata, _ := m["metadata"].(string)
+
+ score := 0.0
+ if additional, ok := m["_additional"].(map[string]interface{}); ok {
+ if certainty, ok := additional["certainty"].(float64); ok {
+ // Weaviate certainty is in [0, 1] where 1 = identical.
+ score = certainty
+ }
+ }
+
+ if e.Config.SimilarityThreshold > 0 && score < e.Config.SimilarityThreshold {
+ log.Debugf("weaviate: filtered out objectID=%s score=%.4f < threshold=%.4f", objectID, score, e.Config.SimilarityThreshold)
+ filtered++
+ continue
+ }
+
+ log.Debugf("weaviate: result objectID=%s objectType=%s score=%.4f", objectID, objectType, score)
+ results = append(results, plugin.VectorSearchResult{
+ ObjectID: objectID,
+ ObjectType: objectType,
+ Metadata: metadata,
+ Score: score,
+ })
+ }
+
+ log.Debugf("weaviate: SearchSimilar done, returning %d results (%d filtered by threshold)", len(results), filtered)
+ return results, nil
+}
+
+// UpdateContent upserts a single document into Weaviate.
+func (e *VectorSearchEngine) UpdateContent(ctx context.Context, content *plugin.VectorSearchContent) error {
+ if e.client == nil {
+ return fmt.Errorf("weaviate client not initialized")
+ }
+
+ log.Debugf("weaviate: UpdateContent objectID=%s objectType=%s title=%q contentLen=%d",
+ content.ObjectID, content.ObjectType, content.Title, len(content.Content))
+
+ log.Debugf("weaviate: generating embedding for objectID=%s via %s model=%s",
+ content.ObjectID, e.Config.EmbeddingAPIHost, e.Config.EmbeddingModel)
+ embedding, err := plugin.GenerateEmbedding(ctx, e.Config.EmbeddingAPIHost, e.Config.EmbeddingAPIKey, e.Config.EmbeddingModel, content.Content)
+ if err != nil {
+ log.Errorf("weaviate: generate embedding for %s failed: %v", content.ObjectID, err)
+ return fmt.Errorf("generate embedding for %s: %w", content.ObjectID, err)
+ }
+ log.Debugf("weaviate: embedding generated for objectID=%s dimensions=%d", content.ObjectID, len(embedding))
+
+ embeddingFloats := make([]float32, len(embedding))
+ copy(embeddingFloats, embedding)
+
+ properties := map[string]interface{}{
+ "objectID": content.ObjectID,
+ "objectType": content.ObjectType,
+ "title": content.Title,
+ "content": content.Content,
+ "metadata": content.Metadata,
+ }
+
+ // Use deterministic UUID so repeated upserts for the same objectID overwrite.
+ uuid := deterministicUUID(content.ObjectID)
+ log.Debugf("weaviate: upsert objectID=%s uuid=%s", content.ObjectID, uuid)
+
+ // Check if object already exists.
+ _, err = e.client.Data().ObjectsGetter().
+ WithClassName(className).
+ WithID(uuid).
+ Do(ctx)
+ if err == nil {
+ // Object exists, update it.
+ log.Debugf("weaviate: updating existing object objectID=%s uuid=%s", content.ObjectID, uuid)
+ err = e.client.Data().Updater().
+ WithClassName(className).
+ WithID(uuid).
+ WithProperties(properties).
+ WithVector(embeddingFloats).
+ Do(ctx)
+ } else {
+ // Object does not exist, create it.
+ log.Debugf("weaviate: creating new object objectID=%s uuid=%s", content.ObjectID, uuid)
+ _, err = e.client.Data().Creator().
+ WithClassName(className).
+ WithProperties(properties).
+ WithVector(embeddingFloats).
+ WithID(uuid).
+ Do(ctx)
+ }
+
+ if err != nil {
+ log.Errorf("weaviate: upsert document %s failed: %v", content.ObjectID, err)
+ return fmt.Errorf("upsert document %s: %w", content.ObjectID, err)
+ }
+ log.Debugf("weaviate: upsert objectID=%s done", content.ObjectID)
+ return nil
+}
+
+// DeleteContent removes a document by object ID.
+func (e *VectorSearchEngine) DeleteContent(ctx context.Context, objectID string) error {
+ if e.client == nil {
+ return fmt.Errorf("weaviate client not initialized")
+ }
+
+ // Delete using deterministic UUID.
+ uuid := deterministicUUID(objectID)
+ log.Debugf("weaviate: DeleteContent objectID=%s uuid=%s", objectID, uuid)
+ err := e.client.Data().Deleter().
+ WithClassName(className).
+ WithID(uuid).
+ Do(ctx)
+ if err != nil {
+ log.Errorf("weaviate: delete document %s failed: %v", objectID, err)
+ return fmt.Errorf("delete document %s: %w", objectID, err)
+ }
+ log.Debugf("weaviate: deleted objectID=%s", objectID)
+ return nil
+}
+
+// ConfigFields returns the plugin configuration form fields.
+func (e *VectorSearchEngine) ConfigFields() []plugin.ConfigField {
+ return []plugin.ConfigField{
+ {
+ Name: "endpoint",
+ Type: plugin.ConfigTypeInput,
+ Title: plugin.MakeTranslator(i18n.ConfigEndpointTitle),
+ Description: plugin.MakeTranslator(i18n.ConfigEndpointDescription),
+ Required: true,
+ UIOptions: plugin.ConfigFieldUIOptions{
+ InputType: plugin.InputTypeText,
+ },
+ Value: e.Config.Endpoint,
+ },
+ {
+ Name: "api_key",
+ Type: plugin.ConfigTypeInput,
+ Title: plugin.MakeTranslator(i18n.ConfigAPIKeyTitle),
+ Description: plugin.MakeTranslator(i18n.ConfigAPIKeyDescription),
+ Required: false,
+ UIOptions: plugin.ConfigFieldUIOptions{
+ InputType: plugin.InputTypePassword,
+ },
+ Value: e.Config.APIKey,
+ },
+ {
+ Name: "embedding_api_host",
+ Type: plugin.ConfigTypeInput,
+ Title: plugin.MakeTranslator(i18n.ConfigEmbeddingAPIHostTitle),
+ Description: plugin.MakeTranslator(i18n.ConfigEmbeddingAPIHostDescription),
+ Required: true,
+ UIOptions: plugin.ConfigFieldUIOptions{
+ InputType: plugin.InputTypeText,
+ },
+ Value: e.Config.EmbeddingAPIHost,
+ },
+ {
+ Name: "embedding_api_key",
+ Type: plugin.ConfigTypeInput,
+ Title: plugin.MakeTranslator(i18n.ConfigEmbeddingAPIKeyTitle),
+ Description: plugin.MakeTranslator(i18n.ConfigEmbeddingAPIKeyDescription),
+ Required: true,
+ UIOptions: plugin.ConfigFieldUIOptions{
+ InputType: plugin.InputTypePassword,
+ },
+ Value: e.Config.EmbeddingAPIKey,
+ },
+ {
+ Name: "embedding_model",
+ Type: plugin.ConfigTypeInput,
+ Title: plugin.MakeTranslator(i18n.ConfigEmbeddingModelTitle),
+ Description: plugin.MakeTranslator(i18n.ConfigEmbeddingModelDescription),
+ Required: true,
+ UIOptions: plugin.ConfigFieldUIOptions{
+ InputType: plugin.InputTypeText,
+ },
+ Value: e.Config.EmbeddingModel,
+ },
+ {
+ Name: "embedding_level",
+ Type: plugin.ConfigTypeSelect,
+ Title: plugin.MakeTranslator(i18n.ConfigEmbeddingLevelTitle),
+ Description: plugin.MakeTranslator(i18n.ConfigEmbeddingLevelDescription),
+ Required: true,
+ Options: []plugin.ConfigFieldOption{
+ {Label: plugin.MakeTranslator(i18n.ConfigEmbeddingLevelOptionQuestion), Value: "question"},
+ {Label: plugin.MakeTranslator(i18n.ConfigEmbeddingLevelOptionAnswer), Value: "answer"},
+ },
+ Value: e.Config.EmbeddingLevel,
+ },
+ {
+ Name: "similarity_threshold",
+ Type: plugin.ConfigTypeInput,
+ Title: plugin.MakeTranslator(i18n.ConfigSimilarityThresholdTitle),
+ Description: plugin.MakeTranslator(i18n.ConfigSimilarityThresholdDescription),
+ Required: false,
+ UIOptions: plugin.ConfigFieldUIOptions{
+ InputType: plugin.InputTypeText,
+ },
+ Value: fmt.Sprintf("%.2f", e.Config.SimilarityThreshold),
+ },
+ }
+}
+
+// ConfigReceiver applies configuration from the admin UI.
+func (e *VectorSearchEngine) ConfigReceiver(config []byte) error {
+ // Pre-process: convert string numbers to actual numbers before unmarshalling,
+ // because the admin UI sends all form values as strings.
+ var raw map[string]interface{}
+ if err := json.Unmarshal(config, &raw); err != nil {
+ return fmt.Errorf("unmarshal config: %w", err)
+ }
+ if threshStr, ok := raw["similarity_threshold"].(string); ok {
+ if thresh, err := strconv.ParseFloat(threshStr, 64); err == nil {
+ raw["similarity_threshold"] = thresh
+ }
+ }
+ fixed, _ := json.Marshal(raw)
+
+ conf := &VectorSearchConfig{}
+ if err := json.Unmarshal(fixed, conf); err != nil {
+ return fmt.Errorf("unmarshal config: %w", err)
+ }
+
+ e.Config = conf
+
+ if !plugin.StatusManager.IsEnabled("weaviate_vector_search") {
+ log.Debugf("weaviate: plugin not active, skipping initialization")
+ return nil
+ }
+
+ log.Debugf("weaviate: initializing client: endpoint=%s scheme=%s host=%s apiKey=%v",
+ conf.Endpoint, parseScheme(conf.Endpoint), stripScheme(conf.Endpoint), conf.APIKey != "")
+ log.Debugf("weaviate: embedding config: host=%s model=%s level=%s threshold=%.2f",
+ conf.EmbeddingAPIHost, conf.EmbeddingModel, conf.EmbeddingLevel, conf.SimilarityThreshold)
+
+ cfg := weaviate.Config{
+ Host: stripScheme(conf.Endpoint),
+ Scheme: parseScheme(conf.Endpoint),
+ }
+ if conf.APIKey != "" {
+ cfg.AuthConfig = auth.ApiKey{Value: conf.APIKey}
+ }
+
+ client, err := weaviate.NewClient(cfg)
+ if err != nil {
+ return fmt.Errorf("create weaviate client: %w", err)
+ }
+
+ // Verify connectivity.
+ log.Debugf("weaviate: checking liveness at %s", conf.Endpoint)
+ live, err := client.Misc().LiveChecker().Do(context.Background())
+ if err != nil || !live {
+ log.Errorf("weaviate: liveness check failed: live=%v err=%v", live, err)
+ return fmt.Errorf("weaviate not reachable at %s: %v", conf.Endpoint, err)
+ }
+ log.Debugf("weaviate: liveness check passed")
+ e.client = client
+
+ // Create/ensure the class with vector configuration.
+ // If it fails with auth error and API key was set, retry without auth (anonymous access).
+ if err := e.ensureClass(context.Background()); err != nil {
+ if conf.APIKey != "" {
+ log.Debugf("weaviate: ensureClass failed with API key auth, retrying without auth (anonymous access)")
+ cfg.AuthConfig = nil
+ client, err = weaviate.NewClient(cfg)
+ if err != nil {
+ return fmt.Errorf("create weaviate client: %w", err)
+ }
+ e.client = client
+ if err := e.ensureClass(context.Background()); err != nil {
+ return fmt.Errorf("ensure class: %w", err)
+ }
+ } else {
+ return fmt.Errorf("ensure class: %w", err)
+ }
+ }
Review Comment:
`ensureClass` failures trigger a retry without API key for any error when an API key is set. That can silently downgrade to anonymous access on transient/network/schema errors, which is risky and makes auth misconfiguration harder to notice. Only retry without auth when the error is clearly auth-related (e.g., 401/403).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@answer.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: add 6 vector-search backend plugins [answer-plugins]
Posted by "LinkinStars (via GitHub)" <gi...@apache.org>.
LinkinStars merged PR #320:
URL: https://github.com/apache/answer-plugins/pull/320
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@answer.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: add 6 vector-search backend plugins [answer-plugins]
Posted by "hgaol (via GitHub)" <gi...@apache.org>.
hgaol commented on PR #320:
URL: https://github.com/apache/answer-plugins/pull/320#issuecomment-4916675996
resolved all comments and validated locally.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@answer.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org