You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hop.apache.org by "bamaer (via GitHub)" <gi...@apache.org> on 2026/09/17 14:37:58 UTC
[I] [Feature Request]: Add an Embed text transform to turn text into embedding vectors (hop)
bamaer opened a new issue, #8441:
URL: https://github.com/apache/hop/issues/8441
### What would you like to happen?
### What would you like to happen?
### Background
Hop can now split documents (#8415) and store vectors in pgvector (#8422), but nothing
in between turns text into a vector. That leaves the retrieval chain broken at its
middle step: chunk, **embed**, store.
The `AiProvider` metadata type from #8330 already models the providers that serve
embedding models, and `AiModelCatalog` can already classify a model as
`ModelType.EMBEDDING`. What is missing is a way to call one.
### Proposal
An `Embed text` transform that reads a text field, calls the embedding model of a
selected `AiProvider`, and adds the vector to the row.
One row in, one row out. The embedding is emitted either as a String holding a JSON
array or as a field of the Vector value type (#8409), so it feeds pgvector upsert
directly in both forms.
Since `AiProvider` has no embedding support yet, this includes extending it to expose
an embedding model alongside the existing chat model, which benefits any later
transform that needs one.
### Options
| Option | Notes |
|---|---|
| AI provider | `AiProvider` metadata object |
| Model | Embedding model name, filtered to `ModelType.EMBEDDING` |
| Input field | Text to embed |
| Output field | Embedding, String or Vector |
| Batch size | Embedding calls are per-request billed and rate limited |
Rows that fail at the provider go to the error hop rather than stopping the pipeline,
so one bad row does not lose a long indexing run.
### Scope
The transform, embedding support on `AiProvider`, unit tests, integration tests against
a local Ollama model, and a user manual page.
Depends on #8330. Composes with #8409 and #8422 but does not require either.
### Issue Priority
Priority: 2
### Issue Component
Component: Transforms
### Issue Priority
Priority: 2
### Issue Component
Component: Transforms
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Request]: Add an Embed text transform to turn text into embedding vectors (hop)
Posted by "bamaer (via GitHub)" <gi...@apache.org>.
bamaer commented on issue #8441:
URL: https://github.com/apache/hop/issues/8441#issuecomment-5718369270
Before I build the transform, a proposal for `AiProvider`, because it changes the shape of this PR.
### The problem
`AiProvider` carries one `modelName`, because it was built for chat. An embedding transform needs a second model from the same endpoint and credentials, and a reranker after a vector search would need a third. langchain4j already separates these: `ChatModel`, `EmbeddingModel`, `ScoringModel`, plus `ImageModel`, `ModerationModel` and the two audio ones.
Without a change, every transform has to carry its own model name, so it gets repeated in every pipeline, or you create one provider object per role and duplicate the base URL and API key in each.
### Proposal: model roles on the provider
```java
// unchanged
private String modelName = ""; // still the chat model
private String temperature = "0.3";
// added
@HopMetadataProperty(key = "models")
private List<AiProviderModel> models = new ArrayList<>();
public class AiProviderModel {
private AiModelRole role; // CHAT, EMBEDDING, SCORING, IMAGE, MODERATION
private String modelName;
}
```
One Ollama provider then covers:
```
CHAT llama3.2
EMBEDDING nomic-embed-text
SCORING bge-reranker-v2-m3
```
A list rather than `embeddingModelName` plus `rerankModelName` plus the next one, so adding a role is a new enum constant instead of a new field, a new editor row and a new migration each time. In the editor it is one `TableView` with a role dropdown and a model name, the same shape as the pgvector column mappings.
### How a transform picks its model
The role is fixed by what the transform is, so the user never chooses one. `Embed text` can only use an `EmbeddingModel`, so it resolves `EMBEDDING`. Language Model Chat resolves `CHAT`. A later rerank transform resolves `SCORING`. Point all three at the same provider object and each gets the right model, with no role dropdown anywhere in a transform dialog.
This means **role is unique within a provider**, one row per role, which is what keeps the lookup unambiguous. A transform needing a different model than the provider's default for its role sets an optional override on the transform itself.
Resolution order for embeddings: transform override, then the provider's `EMBEDDING` row, then `modelName`.
### What changes
- `AiProvider`: the `models` list, a new `AiProviderModel`, an `AiModelRole` enum, and a `resolveModelName(role)` helper.
- `AiProviderEditor`: one `TableView`.
- `AiChatFactory.applyConnection`: one line, `provider.getModelName()` becomes `provider.resolveModelName(CHAT)`, so a `CHAT` row is honoured rather than ignored.
- **Language Model Chat: nothing.** It never reads `AiProvider` fields, it only calls `LanguageModelChatAiProviderSupport.resolve(...)`, which reflects into the factory.
- `AiEmbeddingFactory`: new, alongside `AiChatFactory`, resolving `EMBEDDING`.
### Backward compatibility
Additive, and fully compatible rather than mostly:
- `modelName` and `temperature` keep their current meaning and are not deprecated.
- An existing serialized provider has no `models` key, so it deserializes to an empty list, `CHAT` falls back to `modelName`, and it behaves exactly as it does today.
- A provider edited in a newer Hop and opened in an older one keeps working: the unknown `models` key is ignored and `modelName` is still there.
- There is no serialized `ai-provider` JSON anywhere in the repo today, not even in the integration tests, so there is no stored data to migrate.
- The editor shows an empty table for existing objects, so nothing has to be filled in.
The one case that is not automatic is a user who wants chat and embeddings from one object: they add an `EMBEDDING` row. That is new configuration for new behaviour, not a migration.
### One related gap
`AiModelCatalog.listModelNames` keeps only `CHAT` and `OTHER` and skips the rest, so embedding models are filtered out of the dropdown today. Listing them needs either a type filter on `listModelNames` or a sibling method. To keep this PR small, the model name can be free text with sensible defaults and the dropdown can follow later.
Comments welcome before I build it, since it touches the `AiProvider` metadata type and its editor.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Request]: Add an Embed text transform to turn text into embedding vectors (hop)
Posted by "bamaer (via GitHub)" <gi...@apache.org>.
bamaer closed issue #8441: [Feature Request]: Add an Embed text transform to turn text into embedding vectors
URL: https://github.com/apache/hop/issues/8441
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Request]: Add an Embed text transform to turn text into embedding vectors (hop)
Posted by "bamaer (via GitHub)" <gi...@apache.org>.
bamaer commented on issue #8441:
URL: https://github.com/apache/hop/issues/8441#issuecomment-5716212983
.take-issue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org