You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "xiaochen-zhou (via GitHub)" <gi...@apache.org> on 2025/08/06 16:58:02 UTC
[PR] [Feature][TransForm-V2] Support multimodal embeddings [seatunnel]
xiaochen-zhou opened a new pull request, #9673:
URL: https://github.com/apache/seatunnel/pull/9673
<!--
Thank you for contributing to SeaTunnel! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
## Contribution Checklist
- Make sure that the pull request corresponds to a [GITHUB issue](https://github.com/apache/seatunnel/issues).
- Name the pull request in the form "[Feature] [component] Title of the pull request", where *Feature* can be replaced by `Hotfix`, `Bug`, etc.
- Minor fixes should be named following this pattern: `[hotfix] [docs] Fix typo in README.md doc`.
-->
### Purpose of this pull request
**Multimodal Vectorization:**
```hocon
vectorization_fields {
text_vector = text_field:text # Explicit text type
image_vector = image_url:image # Image modality
video_vector = video_path:video # Video modality
mixed_vector = content_field # Defaults to text if no type specified
}
```
**Field Specification Format:**
- `field_name` - Uses text modality by default
- `field_name:text` - Explicitly specifies text modality
- `field_name:image` - Specifies image modality for image URLs
- `field_name:video` - Specifies video modality for video URLs
### Does this PR introduce _any_ user-facing change?
Yes
### How was this patch tested?
Add new tests
### Check list
* [ ] If any new Jar binary package adding in your PR, please add License Notice according
[New License Guide](https://github.com/apache/seatunnel/blob/dev/docs/en/contribution/new-license.md)
* [ ] If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs
* [ ] If you are contributing the connector code, please check that the following files are updated:
1. Update [plugin-mapping.properties](https://github.com/apache/seatunnel/blob/dev/plugin-mapping.properties) and add new connector information in it
2. Update the pom file of [seatunnel-dist](https://github.com/apache/seatunnel/blob/dev/seatunnel-dist/pom.xml)
3. Add ci label in [label-scope-conf](https://github.com/apache/seatunnel/blob/dev/.github/workflows/labeler/label-scope-conf.yml)
4. Add e2e testcase in [seatunnel-e2e](https://github.com/apache/seatunnel/tree/dev/seatunnel-e2e/seatunnel-connector-v2-e2e/)
5. Update connector [plugin_config](https://github.com/apache/seatunnel/blob/dev/config/plugin_config)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Transform-V2] Support multimodal embeddings [seatunnel]
Posted by "xiaochen-zhou (via GitHub)" <gi...@apache.org>.
xiaochen-zhou commented on code in PR #9673:
URL: https://github.com/apache/seatunnel/pull/9673#discussion_r2312826727
##########
docs/en/transform-v2/embedding.md:
##########
@@ -263,6 +319,165 @@ sink {
}
```
+### Multimodal Embedding (Volcengine Doubao)
Review Comment:
> Could you add an example for binary file mode?
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Transform-V2] Support multimodal embeddings [seatunnel]
Posted by "Hisoka-X (via GitHub)" <gi...@apache.org>.
Hisoka-X commented on code in PR #9673:
URL: https://github.com/apache/seatunnel/pull/9673#discussion_r2312198265
##########
seatunnel-connectors-v2/connector-file/connector-file-base/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/source/reader/BinaryReadStrategy.java:
##########
@@ -49,6 +49,9 @@ public class BinaryReadStrategy extends AbstractReadStrategy {
private boolean completeFileMode =
FileBaseSourceOptions.BINARY_COMPLETE_FILE_MODE.defaultValue();
+ private final String IS_COMPLETE = "is_complete";
+ private final String IS_BINARY_FORMAT = "is_binary_format";
+
Review Comment:
Move it into `CommonOptions`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][TransForm-V2] Support multimodal embeddings [seatunnel]
Posted by "xiaochen-zhou (via GitHub)" <gi...@apache.org>.
xiaochen-zhou commented on PR #9673:
URL: https://github.com/apache/seatunnel/pull/9673#issuecomment-3181465596
> 1. How to get the url base on seatunnel workflow?
> 2. we have binary format in file source, I think we can direct read video or image from file then use base64 send to embedding provider. https://www.volcengine.com/docs/82379/1409291#base64-%E7%BC%96%E7%A0%81%E8%BE%93%E5%85%A5
OK.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Transform-V2] Support multimodal embeddings [seatunnel]
Posted by "Hisoka-X (via GitHub)" <gi...@apache.org>.
Hisoka-X commented on code in PR #9673:
URL: https://github.com/apache/seatunnel/pull/9673#discussion_r2312721358
##########
docs/en/transform-v2/embedding.md:
##########
@@ -263,6 +319,165 @@ sink {
}
```
+### Multimodal Embedding (Volcengine Doubao)
Review Comment:
Could you add an example for binary file mode?
##########
seatunnel-connectors-v2/connector-file/connector-file-base/src/test/java/org/apache/seatunnel/connectors/seatunnel/file/reader/BinaryReadStrategyTest.java:
##########
@@ -70,7 +70,9 @@ public void testBinaryReadWithDefaultChunkSize() throws IOException {
List<SeaTunnelRow> rows = collector.getRows();
Assertions.assertEquals(
- 2, rows.size(), "Should have 2 chunks for 2048 bytes with default 1024 chunk size");
+ 2 + 1,
+ rows.size(),
+ "Should have 2 chunks for 2048 bytes with default 1024 chunk size");
Review Comment:
```suggestion
"Should have 3 chunks for 2048 bytes with default 1024 chunk size");
```
##########
seatunnel-connectors-v2/connector-file/connector-file-base/src/test/java/org/apache/seatunnel/connectors/seatunnel/file/reader/BinaryReadStrategyTest.java:
##########
@@ -128,7 +130,7 @@ public void testBinaryReadCompleteFileMode() throws IOException {
binaryReadStrategy.read(testFile.getAbsolutePath(), "test_table", collector);
List<SeaTunnelRow> rows = collector.getRows();
- Assertions.assertEquals(1, rows.size(), "Should have 1 row in complete file mode");
+ Assertions.assertEquals(1 + 1, rows.size(), "Should have 1 row in complete file mode");
Review Comment:
```suggestion
Assertions.assertEquals(1 + 1, rows.size(), "Should have 2 row in complete file mode");
```
##########
seatunnel-connectors-v2/connector-file/connector-file-base/src/test/java/org/apache/seatunnel/connectors/seatunnel/file/reader/BinaryReadStrategyTest.java:
##########
@@ -102,7 +104,7 @@ public void testBinaryReadWithCustomChunkSize() throws IOException {
List<SeaTunnelRow> rows = collector.getRows();
Assertions.assertEquals(
- 3, rows.size(), "Should have 3 chunks for 1500 bytes with 512 chunk size");
+ 3 + 1, rows.size(), "Should have 3 chunks for 1500 bytes with 512 chunk size");
Review Comment:
```suggestion
3 + 1, rows.size(), "Should have 4 chunks for 1500 bytes with 512 chunk size");
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Transform-V2] Support multimodal embeddings [seatunnel]
Posted by "corgy-w (via GitHub)" <gi...@apache.org>.
corgy-w merged PR #9673:
URL: https://github.com/apache/seatunnel/pull/9673
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][TransForm-V2] Support multimodal embeddings [seatunnel]
Posted by "xiaochen-zhou (via GitHub)" <gi...@apache.org>.
xiaochen-zhou commented on code in PR #9673:
URL: https://github.com/apache/seatunnel/pull/9673#discussion_r2271619977
##########
seatunnel-transforms-v2/src/main/java/org/apache/seatunnel/transform/nlpmodel/embedding/multimodal/MultimodalField.java:
##########
@@ -0,0 +1,55 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.seatunnel.transform.nlpmodel.embedding.multimodal;
+
+import lombok.AllArgsConstructor;
+import lombok.Getter;
+import lombok.ToString;
+
+@Getter
+@AllArgsConstructor
+@ToString
+public class MultimodalField {
+
+ private String fieldName;
+ private ModalityType modalityType;
+
+ public MultimodalField(String fieldSpec) {
+ if (fieldSpec == null || fieldSpec.trim().isEmpty()) {
+ throw new IllegalArgumentException("Field specification cannot be null or empty");
+ }
+
+ String trimmedSpec = fieldSpec.trim();
+ if (trimmedSpec.contains(":")) {
Review Comment:
> When creating columns in the database, it is possible to add `:`, but is it incompatible here
```
vectorization_fields {
text_vector {
field = "text_field"
modality = "text"
}
image_vector {
field = "image_data"
modality = "image"
}
}
```
To ensure compatibility in this situation, what do you think about this format?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][TransForm-V2] Support multimodal embeddings [seatunnel]
Posted by "xiaochen-zhou (via GitHub)" <gi...@apache.org>.
xiaochen-zhou commented on code in PR #9673:
URL: https://github.com/apache/seatunnel/pull/9673#discussion_r2271619977
##########
seatunnel-transforms-v2/src/main/java/org/apache/seatunnel/transform/nlpmodel/embedding/multimodal/MultimodalField.java:
##########
@@ -0,0 +1,55 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.seatunnel.transform.nlpmodel.embedding.multimodal;
+
+import lombok.AllArgsConstructor;
+import lombok.Getter;
+import lombok.ToString;
+
+@Getter
+@AllArgsConstructor
+@ToString
+public class MultimodalField {
+
+ private String fieldName;
+ private ModalityType modalityType;
+
+ public MultimodalField(String fieldSpec) {
+ if (fieldSpec == null || fieldSpec.trim().isEmpty()) {
+ throw new IllegalArgumentException("Field specification cannot be null or empty");
+ }
+
+ String trimmedSpec = fieldSpec.trim();
+ if (trimmedSpec.contains(":")) {
Review Comment:
> When creating columns in the database, it is possible to add `:`, but is it incompatible here
```
vectorization_fields {
text_vector {
field = "text_field"
modality = "text"
format= "binary"
}
image_vector {
field = "image_data"
modality = "image"
format= "url"
}
}
```
To ensure compatibility in this situation, what do you think about this format?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Transform-V2] Support multimodal embeddings [seatunnel]
Posted by "xiaochen-zhou (via GitHub)" <gi...@apache.org>.
xiaochen-zhou commented on code in PR #9673:
URL: https://github.com/apache/seatunnel/pull/9673#discussion_r2312781362
##########
docs/en/transform-v2/embedding.md:
##########
@@ -263,6 +319,165 @@ sink {
}
```
+### Multimodal Embedding (Volcengine Doubao)
Review Comment:
> Could you add an example for binary file mode?
OK.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][TransForm-V2] Support multimodal embeddings [seatunnel]
Posted by "xiaochen-zhou (via GitHub)" <gi...@apache.org>.
xiaochen-zhou commented on code in PR #9673:
URL: https://github.com/apache/seatunnel/pull/9673#discussion_r2271585117
##########
docs/en/transform-v2/embedding.md:
##########
@@ -19,7 +19,7 @@ different API endpoints.
| secret_key | string | yes | - | The secret key required for additional authentication with the embedding service. |
| aws_region | string | no | | AWS Region. Required for use Amazon Bedrock model. |
| single_vectorized_input_number | int | no | 1 | The number of inputs vectorized in one request. Default is 1. |
-| vectorization_fields | map | yes | - | A mapping between input fields and their corresponding output vector fields. |
+| vectorization_fields | map | yes | - | A mapping between input fields and their corresponding output vector fields. Supports multimodal field specifications with format `field_name:modality_type`. |
| model | string | yes | - | The specific model to use for embedding (e.g: `text-embedding-3-small` for OPENAI). |
| api_path | string | no | - | The API endpoint for the embedding service. Typically provided by the model provider. |
| dimension | int | no | - | TThe vector dimension defaults to 2048. The Embedding-3 model supports custom vector dimensions, and it is recommended to choose dimensions of 256, 512, 1024, or 2048. |
Review Comment:
> Please help modify it. There's an extra "T" here
OK.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][TransForm-V2] Support multimodal embeddings [seatunnel]
Posted by "xiaochen-zhou (via GitHub)" <gi...@apache.org>.
xiaochen-zhou commented on PR #9673:
URL: https://github.com/apache/seatunnel/pull/9673#issuecomment-3194785762
@Hisoka-X @corgy-w Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][TransForm-V2] Support multimodal embeddings [seatunnel]
Posted by "corgy-w (via GitHub)" <gi...@apache.org>.
corgy-w commented on code in PR #9673:
URL: https://github.com/apache/seatunnel/pull/9673#discussion_r2258701392
##########
docs/en/transform-v2/embedding.md:
##########
@@ -19,7 +19,7 @@ different API endpoints.
| secret_key | string | yes | - | The secret key required for additional authentication with the embedding service. |
| aws_region | string | no | | AWS Region. Required for use Amazon Bedrock model. |
| single_vectorized_input_number | int | no | 1 | The number of inputs vectorized in one request. Default is 1. |
-| vectorization_fields | map | yes | - | A mapping between input fields and their corresponding output vector fields. |
+| vectorization_fields | map | yes | - | A mapping between input fields and their corresponding output vector fields. Supports multimodal field specifications with format `field_name:modality_type`. |
| model | string | yes | - | The specific model to use for embedding (e.g: `text-embedding-3-small` for OPENAI). |
| api_path | string | no | - | The API endpoint for the embedding service. Typically provided by the model provider. |
| dimension | int | no | - | TThe vector dimension defaults to 2048. The Embedding-3 model supports custom vector dimensions, and it is recommended to choose dimensions of 256, 512, 1024, or 2048. |
Review Comment:
Please help modify it. There's an extra "T" here
##########
seatunnel-transforms-v2/src/main/java/org/apache/seatunnel/transform/nlpmodel/embedding/multimodal/MultimodalField.java:
##########
@@ -0,0 +1,55 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.seatunnel.transform.nlpmodel.embedding.multimodal;
+
+import lombok.AllArgsConstructor;
+import lombok.Getter;
+import lombok.ToString;
+
+@Getter
+@AllArgsConstructor
+@ToString
+public class MultimodalField {
+
+ private String fieldName;
+ private ModalityType modalityType;
+
+ public MultimodalField(String fieldSpec) {
+ if (fieldSpec == null || fieldSpec.trim().isEmpty()) {
+ throw new IllegalArgumentException("Field specification cannot be null or empty");
+ }
+
+ String trimmedSpec = fieldSpec.trim();
+ if (trimmedSpec.contains(":")) {
Review Comment:
When creating columns in the database, it is possible to add `:`, but is it incompatible here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][TransForm-V2] Support multimodal embeddings [seatunnel]
Posted by "Hisoka-X (via GitHub)" <gi...@apache.org>.
Hisoka-X commented on PR #9673:
URL: https://github.com/apache/seatunnel/pull/9673#issuecomment-3162237013
1. How to get the url base on seatunnel workflow?
2. we have binary format in file source, I think we can direct read video or image from file then use base64 send to embedding provider. https://www.volcengine.com/docs/82379/1409291#base64-%E7%BC%96%E7%A0%81%E8%BE%93%E5%85%A5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Transform-V2] Support multimodal embeddings [seatunnel]
Posted by "xiaochen-zhou (via GitHub)" <gi...@apache.org>.
xiaochen-zhou commented on PR #9673:
URL: https://github.com/apache/seatunnel/pull/9673#issuecomment-3247400031
> Overall LGTM. Waiting test case passes.
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Transform-V2] Support multimodal embeddings [seatunnel]
Posted by "Hisoka-X (via GitHub)" <gi...@apache.org>.
Hisoka-X commented on code in PR #9673:
URL: https://github.com/apache/seatunnel/pull/9673#discussion_r2297274241
##########
seatunnel-e2e/seatunnel-transforms-v2-e2e/seatunnel-transforms-v2-e2e-part-1/src/test/resources/embedding_transform_binary.conf:
##########
@@ -0,0 +1,88 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+env {
+ job.mode = "BATCH"
+}
+
+source {
+ LocalFile {
+ path = "/seatunnel/read/binary/"
+ file_format_type = "binary"
+ binary_complete_file_mode = false
+ binary_chunk_size = 1024
+ schema {
+ table = "cat"
+ }
Review Comment:
what's for?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][TransForm-V2] Support multimodal embeddings [seatunnel]
Posted by "Hisoka-X (via GitHub)" <gi...@apache.org>.
Hisoka-X commented on PR #9673:
URL: https://github.com/apache/seatunnel/pull/9673#issuecomment-3182031998
> > 1. How to get the url base on seatunnel workflow?
> > 2. we have binary format in file source, I think we can direct read video or image from file then use base64 send to embedding provider. https://www.volcengine.com/docs/82379/1409291#base64-%E7%BC%96%E7%A0%81%E8%BE%93%E5%85%A5
>
> Sometimes users may directly store image or video paths in the columns. Should we consider both scenarios, or just focus on one of them?
>
> ```
> vectorization_fields {
> text_vector {
> field = "text_field"
> modality = "text"
> format= "binary"
>
> }
> image_vector {
> field = "image_data"
> modality = "image"
> format= "url"
> }
> }
> ```
both.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][TransForm-V2] Support multimodal embeddings [seatunnel]
Posted by "xiaochen-zhou (via GitHub)" <gi...@apache.org>.
xiaochen-zhou commented on PR #9673:
URL: https://github.com/apache/seatunnel/pull/9673#issuecomment-3181610625
> 1. How to get the url base on seatunnel workflow?
> 2. we have binary format in file source, I think we can direct read video or image from file then use base64 send to embedding provider. https://www.volcengine.com/docs/82379/1409291#base64-%E7%BC%96%E7%A0%81%E8%BE%93%E5%85%A5
Sometimes users may directly store image or video paths in the columns. Should we consider both scenarios, or just focus on one of them?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][TransForm-V2] Support multimodal embeddings [seatunnel]
Posted by "Hisoka-X (via GitHub)" <gi...@apache.org>.
Hisoka-X commented on code in PR #9673:
URL: https://github.com/apache/seatunnel/pull/9673#discussion_r2284318174
##########
seatunnel-transforms-v2/src/main/java/org/apache/seatunnel/transform/nlpmodel/embedding/EmbeddingTransform.java:
##########
@@ -180,33 +187,49 @@ public void open() {
}
}
- private void initOutputFields(SeaTunnelRowType inputRowType, Map<String, String> fields) {
+ private void initOutputFields(SeaTunnelRowType inputRowType, ReadonlyConfig config) {
+ Map<Integer, FieldSpec> fieldSpecMap = new HashMap<>();
List<String> fieldNames = new ArrayList<>();
- List<Integer> fieldOriginalIndexes = new ArrayList<>();
- for (Map.Entry<String, String> field : fields.entrySet()) {
- String srcField = field.getValue();
+ Map<String, Object> fieldsConfig =
+ config.get(EmbeddingTransformConfig.VECTORIZATION_FIELDS);
+ if (fieldsConfig == null || fieldsConfig.isEmpty()) {
+ throw new IllegalArgumentException("vectorization_fields configuration is required");
+ }
+
+ for (Map.Entry<String, Object> field : fieldsConfig.entrySet()) {
+ FieldSpec fieldSpec = new FieldSpec(field);
+ String srcField = fieldSpec.getFieldName();
int srcFieldIndex;
try {
srcFieldIndex = inputRowType.indexOf(srcField);
} catch (IllegalArgumentException e) {
throw TransformCommonError.cannotFindInputFieldError(getPluginName(), srcField);
}
+ if (fieldSpec.isMultimodalField()) {
+ isMultimodalFields = true;
+ }
+ fieldSpecMap.put(srcFieldIndex, fieldSpec);
fieldNames.add(field.getKey());
- fieldOriginalIndexes.add(srcFieldIndex);
}
+ this.fieldSpecMap = fieldSpecMap;
this.fieldNames = fieldNames;
- this.fieldOriginalIndexes = fieldOriginalIndexes;
}
@Override
protected Object[] getOutputFieldValues(SeaTunnelRowAccessor inputRow) {
tryOpen();
try {
- Object[] fieldArray = new Object[fieldOriginalIndexes.size()];
- for (int i = 0; i < fieldOriginalIndexes.size(); i++) {
- fieldArray[i] = inputRow.getField(fieldOriginalIndexes.get(i));
+ Set<Integer> fieldOriginalIndexes = fieldSpecMap.keySet();
+ Object[] fieldValues = new Object[fieldOriginalIndexes.size()];
+ List<ByteBuffer> vectorization;
+ int i = 0;
+ for (Integer fieldOriginalIndex : fieldOriginalIndexes) {
+ FieldSpec fieldSpec = fieldSpecMap.get(fieldOriginalIndex);
+ Object value = inputRow.getField(fieldOriginalIndex);
+ fieldValues[i++] =
+ isMultimodalFields ? new MultimodalFieldValue(fieldSpec, value) : value;
Review Comment:
In the binary format scenario, the data contained in each row may be part of the complete file. We need to cache it in the transform and perform embedding after obtaining the complete file. Please refer to https://github.com/apache/seatunnel/blob/dev/seatunnel-connectors-v2/connector-file/connector-file-base/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/source/reader/BinaryReadStrategy.java#L132 and https://github.com/apache/seatunnel/blob/dev/seatunnel-connectors-v2/connector-file/connector-file-base/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/sink/writer/BinaryWriteStrategy.java#L78
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Transform-V2] Support multimodal embeddings [seatunnel]
Posted by "Hisoka-X (via GitHub)" <gi...@apache.org>.
Hisoka-X commented on PR #9673:
URL: https://github.com/apache/seatunnel/pull/9673#issuecomment-3242504232
Overall LGTM. Waiting test case passes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Transform-V2] Support multimodal embeddings [seatunnel]
Posted by "Hisoka-X (via GitHub)" <gi...@apache.org>.
Hisoka-X commented on code in PR #9673:
URL: https://github.com/apache/seatunnel/pull/9673#discussion_r2312195646
##########
seatunnel-api/src/main/java/org/apache/seatunnel/api/table/type/SeaTunnelRow.java:
##########
@@ -126,6 +126,14 @@ public int getBytesSize(SeaTunnelRowType rowType) {
return size;
}
+ public boolean isBinaryFormat() {
+ return getOptions().getOrDefault("is_binary_format", false).equals(true);
+ }
+
+ public boolean isComplete() {
+ return getOptions().getOrDefault("is_complete", false).equals(true);
+ }
Review Comment:
The option value belong binary format, so I think we should not put the method into `SeaTunnelRow`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org