You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "hailin0 (via GitHub)" <gi...@apache.org> on 2024/07/14 08:06:15 UTC
[I] [Feature][Weaviate] Support weaviate connector source/sink [seatunnel]
hailin0 opened a new issue, #7192:
URL: https://github.com/apache/seatunnel/issues/7192
### Search before asking
- [X] I had searched in the [feature](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22Feature%22) and found no similar feature requirement.
### Description
Currently, we already support the milvus connector, would anyone like to add support for weaviate?
reference
https://github.com/weaviate/weaviate
https://weaviate.io/developers/weaviate/client-libraries/java
milvus connector pr
https://github.com/apache/seatunnel/pull/7158
### Usage Scenario
_No response_
### Related issues
_No response_
### Are you willing to submit a PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature][Weaviate] Support weaviate connector source/sink [seatunnel]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #7192: [Feature][Weaviate] Support weaviate connector source/sink
URL: https://github.com/apache/seatunnel/issues/7192
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature][Weaviate] Support weaviate connector source/sink [seatunnel]
Posted by "corgy-w (via GitHub)" <gi...@apache.org>.
corgy-w commented on issue #7192:
URL: https://github.com/apache/seatunnel/issues/7192#issuecomment-2351419854
# Weaviate
## Data Format
Let's consider the vector data format from Milvus as follows:
| Field | Type | Description |
| -------------- | ------------ | ----------- |
| book_name | string | andersen |
| book_id | int64 | 1 |
| summary | string | fairy tale |
| summary_vector | float_vector | ..... |
| name_vector | float_vector | ..... |
When synchronizing to Weaviate, several challenges arise.
First, let’s refer to the [Weaviate data format](https://weaviate.io/developers/weaviate/manage-data/collections):
```
Schema
├── Class 1
│ ├── Property 1
│ ├── Property 2
│ └── Object/Instance 1, Object/Instance 2, ...
├── Class 2
│ ├── Property 1
│ ├── Property 2
│ └── Object/Instance 1, Object/Instance 2, ...
└── Class 3
├── Property 1
├── Property 2
└── Object/Instance 1, Object/Instance 2, ...
```
In this mapping:
- Milvus table name corresponds to the Weaviate class name.
- Milvus field name corresponds to the Weaviate property name.
Vectors in Weaviate are treated as class properties, where a class can have one vector field representing a specific attribute.
### Preliminary Design:
1. **As Sink**
1. **Data without existing vector fields**:
Weaviate's vector database has an embedded OpenAI vector transformation engine, allowing you to specify a field to automatically generate vectors, which can then be synchronized into Weaviate.
2. **Data with one vector field**:
You can manually specify which field is the primary attribute and which one holds the vector data.
Example:
If `book_name` is the main field and `name_vector` is the vector field:
```java
float[] customVector = {0.1f, 0.2f, 0.3f};
object.set(bookName, true); // true indicates the vector field
object.setVector(customVector); // manually set the vector, bypassing Weaviate auto-vectorization
```
3. **Data with multiple vector fields**:
In cases where there are two or more vector fields, it’s necessary to ignore additional vectors beyond the configured field.
2. **As Source**
1. You need to specify which field within the class corresponds to the primary vector field; otherwise, data retrieval won’t work as expected.
The vector field name should be configured appropriately for data access.
---
## Application Scenarios
**Weaviate for fast embedded vector processing, Milvus for large-scale vector management**: Weaviate provides built-in vectorizers (such as transformers, Cohere, OpenAI) that automatically convert text, images, and other data into vectors, and manage the entire process of storage and querying. This makes it especially suited for small-scale data analysis and embedded vector processing. Projects that require a quick setup and simplified data processing workflows benefit from Weaviate’s all-in-one approach, allowing easy implementation of embedding transformations.**Therefore, I prefer the first use of sink**
On the other hand, **Milvus excels in managing large-scale vector data**. It delegates the vector generation process to external tools, allowing users greater flexibility in controlling vector creation and insertion, supporting efficient vector retrieval and complex queries. This makes Milvus ideal for handling large volumes of vector data, where custom control over the data pipeline is essential.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature][Weaviate] Support weaviate connector source/sink [seatunnel]
Posted by "corgy-w (via GitHub)" <gi...@apache.org>.
corgy-w commented on issue #7192:
URL: https://github.com/apache/seatunnel/issues/7192#issuecomment-2343271639
Can be assigned to me temporarily
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature][Weaviate] Support weaviate connector source/sink [seatunnel]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #7192: [Feature][Weaviate] Support weaviate connector source/sink
URL: https://github.com/apache/seatunnel/issues/7192
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature][Weaviate] Support weaviate connector source/sink [seatunnel]
Posted by "Hisoka-X (via GitHub)" <gi...@apache.org>.
Hisoka-X commented on issue #7192:
URL: https://github.com/apache/seatunnel/issues/7192#issuecomment-2357422816
Thanks @corgy-w for share these important information!
As Sink:
I think we should let users decide how their data is used, and we can add parameters to control the behavior.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature][Weaviate] Support weaviate connector source/sink [seatunnel]
Posted by "corgy-w (via GitHub)" <gi...@apache.org>.
corgy-w commented on issue #7192:
URL: https://github.com/apache/seatunnel/issues/7192#issuecomment-2351436338
Feels like the storage format of `pinecone` are similar to Weaviate
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature][Weaviate] Support weaviate connector source/sink [seatunnel]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #7192:
URL: https://github.com/apache/seatunnel/issues/7192#issuecomment-2287491712
This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature][Weaviate] Support weaviate connector source/sink [seatunnel]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #7192:
URL: https://github.com/apache/seatunnel/issues/7192#issuecomment-3092662274
This issue has been closed because it has not received response for too long time. You could reopen it if you encountered similar problems in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature][Weaviate] Support weaviate connector source/sink [seatunnel]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #7192:
URL: https://github.com/apache/seatunnel/issues/7192#issuecomment-2327681297
This issue has been closed because it has not received response for too long time. You could reopen it if you encountered similar problems in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature][Weaviate] Support weaviate connector source/sink [seatunnel]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #7192:
URL: https://github.com/apache/seatunnel/issues/7192#issuecomment-3064432966
This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org