You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "ss666 (via GitHub)" <gi...@apache.org> on 2026/07/12 13:34:53 UTC
[PR] [Feature][Transform-V2] Add TextChunk transform plugin [seatunnel]
ss666 opened a new pull request, #11414:
URL: https://github.com/apache/seatunnel/pull/11414
### Purpose of this pull request
Closes #9717.
Adds a new `TextChunk` plugin that splits a long text field into multiple smaller chunks for RAG / embedding pipelines, emitting one row per chunk (a 1-row → N-rows FlatMap). Each output row keeps all original source fields and appends two columns: the chunk text (`chunk`) and its 0-based index within the source document (`chunk_index`).
Main points:
1. `TextChunker`: a pure, dependency-free chunking algorithm. It splits recursively by a priority list of `separators`, then greedily merges pieces up to `chunk_size`; when `separators` is empty it falls back to fixed-size windows. Overlap between adjacent chunks is carried as whole trailing pieces (never mid-word) in the separator path; boundaries are aligned to whole Unicode code points so a surrogate pair (e.g. an emoji) is never split.
2. `TextChunkTransform` / `TextChunkTransformFactory` / `TextChunkMultiCatalogTransform`: the SPI wiring — options, check-time validation, schema extension, and multi-catalog dispatch.
3. Options: `text_field`, `output_field`, `chunk_index_field`, `chunk_size`, `overlap_size`, `separators`.
4. Docs (`docs/en` + `docs/zh`) and unit/E2E tests added.
### Does this PR introduce _any_ user-facing change?
Yes — a new optional transform plugin `TextChunk`. It is purely additive: no existing option key, default, public API or SPI contract is changed, so existing jobs are unaffected. New user-facing docs are added under `docs/en/transforms/text-chunk.md` and `docs/zh/transforms/text-chunk.md`.
### How was this patch tested?
Scoped to the changed module:
```bash
./mvnw spotless:apply -pl seatunnel-transforms-v2,seatunnel-e2e/seatunnel-transforms-v2-e2e/seatunnel-transforms-v2-e2e-part-1
./mvnw -DskipTests verify -pl seatunnel-transforms-v2
./mvnw test -pl seatunnel-transforms-v2 \
-Dtest='TextChunkerTest,TextChunkTransformTest,TextChunkTransformFactoryTest'
```
E2E: two configs added — `text_chunk_transform.conf` (single table; asserts exact chunk content and index bounds) and `text_chunk_transform_multi_table.conf` (per-table override + top-level regex fallback).
### Check list
* [ ] If any new Jar binary package adding in your PR, please add License Notice according [New License Guide](https://github.com/apache/seatunnel/blob/dev/docs/en/contribution/new-license.md) — *N/A: no new jar/dependency.*
* [x] If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs — *Added `docs/en` + `docs/zh` `text-chunk.md` and registered in `docs/sidebars.js`.*
* [ ] If necessary, please update `incompatible-changes.md` to describe the incompatibility caused by this PR. — *N/A: purely additive, no incompatibility.*
* [ ] If you are contributing the connector code, please check that the following files are updated: — *N/A: this is a Transform-V2 plugin, not a connector. It registers via `@AutoService` (SPI), and `seatunnel-transforms-v2` is already in `seatunnel-dist`, so `plugin-mapping.properties`, the dist pom, and `config/plugin_config` need no change.*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Transform-V2] Add TextChunk transform plugin [seatunnel]
Posted by "nzw921rx (via GitHub)" <gi...@apache.org>.
nzw921rx commented on code in PR #11414:
URL: https://github.com/apache/seatunnel/pull/11414#discussion_r3567839005
##########
seatunnel-transforms-v2/src/main/java/org/apache/seatunnel/transform/chunk/TextChunkTransformConfig.java:
##########
@@ -0,0 +1,119 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.seatunnel.transform.chunk;
+
+import org.apache.seatunnel.api.configuration.Option;
+import org.apache.seatunnel.api.configuration.Options;
+import org.apache.seatunnel.api.configuration.ReadonlyConfig;
+import org.apache.seatunnel.api.configuration.util.OptionValidationException;
+
+import lombok.Getter;
+import lombok.Setter;
+
+import java.io.Serializable;
+import java.util.Arrays;
+import java.util.List;
+
+@Getter
+@Setter
+public class TextChunkTransformConfig implements Serializable {
+
+ public static final Option<String> TEXT_FIELD =
+ Options.key("text_field")
+ .stringType()
+ .noDefaultValue()
+ .withDescription("The source text field to split into chunks");
+
+ public static final Option<String> OUTPUT_FIELD =
+ Options.key("output_field")
+ .stringType()
+ .defaultValue("chunk")
+ .withDescription("The output field name holding each chunk (STRING)");
+
+ public static final Option<String> CHUNK_INDEX_FIELD =
+ Options.key("chunk_index_field")
+ .stringType()
+ .defaultValue("chunk_index")
+ .withDescription(
+ "The output field name holding the chunk sequence index within a "
+ + "document (INT, 0-based)");
+
+ public static final Option<Integer> CHUNK_SIZE =
+ Options.key("chunk_size")
+ .intType()
+ .defaultValue(1000)
+ .withDescription("Maximum length of each chunk, measured in characters");
+
+ public static final Option<Integer> OVERLAP_SIZE =
+ Options.key("overlap_size")
+ .intType()
+ .defaultValue(0)
+ .withDescription(
+ "Overlap length between adjacent chunks, measured in characters. "
+ + "Must be less than chunk_size");
+
+ public static final Option<List<String>> SEPARATORS =
+ Options.key("separators")
+ .listType()
+ .defaultValue(Arrays.asList("\n\n", "\n", "。", "!", "?", ". ", " "))
+ .withDescription(
+ "Separators tried in priority order to avoid cutting mid-sentence. "
+ + "If left empty, falls back to fixed-size splitting");
+
+ private String textField;
+ private String outputField;
+ private String chunkIndexField;
+ private int chunkSize;
+ private int overlapSize;
+ private List<String> separators;
+
+ public static TextChunkTransformConfig of(ReadonlyConfig config) {
+ TextChunkTransformConfig c = new TextChunkTransformConfig();
+ c.setTextField(config.get(TEXT_FIELD));
+ c.setOutputField(config.get(OUTPUT_FIELD));
+ c.setChunkIndexField(config.get(CHUNK_INDEX_FIELD));
+ c.setChunkSize(config.get(CHUNK_SIZE));
+ c.setOverlapSize(config.get(OVERLAP_SIZE));
+ c.setSeparators(config.get(SEPARATORS));
+ return c;
+ }
+
+ public static void validate(TextChunkTransformConfig config) {
Review Comment:
Thank you for your contribution. This is an essential supplement to RAG. Currently,
the `validate (TextChunkTransformConfig config) ` function is used by Factory and has no common use. In terms of the scope of responsibilities of the class, this function is more suitable to be placed in the `TextChunkTransformFactory`. If you have any different opinions, please feel free to discuss
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Transform-V2] Add TextChunk transform plugin [seatunnel]
Posted by "davidzollo (via GitHub)" <gi...@apache.org>.
davidzollo merged PR #11414:
URL: https://github.com/apache/seatunnel/pull/11414
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Transform-V2] Add TextChunk transform plugin [seatunnel]
Posted by "nzw921rx (via GitHub)" <gi...@apache.org>.
nzw921rx commented on code in PR #11414:
URL: https://github.com/apache/seatunnel/pull/11414#discussion_r3567839005
##########
seatunnel-transforms-v2/src/main/java/org/apache/seatunnel/transform/chunk/TextChunkTransformConfig.java:
##########
@@ -0,0 +1,119 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.seatunnel.transform.chunk;
+
+import org.apache.seatunnel.api.configuration.Option;
+import org.apache.seatunnel.api.configuration.Options;
+import org.apache.seatunnel.api.configuration.ReadonlyConfig;
+import org.apache.seatunnel.api.configuration.util.OptionValidationException;
+
+import lombok.Getter;
+import lombok.Setter;
+
+import java.io.Serializable;
+import java.util.Arrays;
+import java.util.List;
+
+@Getter
+@Setter
+public class TextChunkTransformConfig implements Serializable {
+
+ public static final Option<String> TEXT_FIELD =
+ Options.key("text_field")
+ .stringType()
+ .noDefaultValue()
+ .withDescription("The source text field to split into chunks");
+
+ public static final Option<String> OUTPUT_FIELD =
+ Options.key("output_field")
+ .stringType()
+ .defaultValue("chunk")
+ .withDescription("The output field name holding each chunk (STRING)");
+
+ public static final Option<String> CHUNK_INDEX_FIELD =
+ Options.key("chunk_index_field")
+ .stringType()
+ .defaultValue("chunk_index")
+ .withDescription(
+ "The output field name holding the chunk sequence index within a "
+ + "document (INT, 0-based)");
+
+ public static final Option<Integer> CHUNK_SIZE =
+ Options.key("chunk_size")
+ .intType()
+ .defaultValue(1000)
+ .withDescription("Maximum length of each chunk, measured in characters");
+
+ public static final Option<Integer> OVERLAP_SIZE =
+ Options.key("overlap_size")
+ .intType()
+ .defaultValue(0)
+ .withDescription(
+ "Overlap length between adjacent chunks, measured in characters. "
+ + "Must be less than chunk_size");
+
+ public static final Option<List<String>> SEPARATORS =
+ Options.key("separators")
+ .listType()
+ .defaultValue(Arrays.asList("\n\n", "\n", "。", "!", "?", ". ", " "))
+ .withDescription(
+ "Separators tried in priority order to avoid cutting mid-sentence. "
+ + "If left empty, falls back to fixed-size splitting");
+
+ private String textField;
+ private String outputField;
+ private String chunkIndexField;
+ private int chunkSize;
+ private int overlapSize;
+ private List<String> separators;
+
+ public static TextChunkTransformConfig of(ReadonlyConfig config) {
+ TextChunkTransformConfig c = new TextChunkTransformConfig();
+ c.setTextField(config.get(TEXT_FIELD));
+ c.setOutputField(config.get(OUTPUT_FIELD));
+ c.setChunkIndexField(config.get(CHUNK_INDEX_FIELD));
+ c.setChunkSize(config.get(CHUNK_SIZE));
+ c.setOverlapSize(config.get(OVERLAP_SIZE));
+ c.setSeparators(config.get(SEPARATORS));
+ return c;
+ }
+
+ public static void validate(TextChunkTransformConfig config) {
Review Comment:
Thank you for your contribution. This is an essential supplement to RAG. Currently,
the `validate (TextChunkTransformConfig config) ` function is used by Factory and has no common use. From the perspective of class self blame, this function is more suitable to be placed in `TextChunkTransformFactory`. If you have any different opinions, please feel free to discuss
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Transform-V2] Add TextChunk transform plugin [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11414:
URL: https://github.com/apache/seatunnel/pull/11414#issuecomment-4968432879
Rechecked the same unchanged head `a3fc770b6be1` after the CI / merge-gate fact changed.
## Runtime path rechecked
```text
Job config
-> TextChunkTransformFactory.optionRule()
-> validate text_field / chunk_size / overlap_size / output field names
-> TextChunkTransformFactory.createTransform()
-> new TextChunkMultiCatalogTransform(catalogTables, options)
Schema production
-> TextChunkMultiCatalogTransform.buildTransform()
-> new TextChunkTransform(config, inputCatalogTable)
-> locate text_field in the input row type
-> transformTableSchema()
-> copy original columns
-> add or replace output_field
-> add or replace chunk_index_field
-> extend normal primary keys and unique keys with chunk_index_field
Row processing
-> TextChunkTransform.flatMap(inputRow)
-> transformRow(inputRow)
-> null / empty text: emit no rows
-> TextChunker.split(...)
-> copy source fields for each chunk
-> write chunk and chunk_index
-> preserve tableId / rowKind / row options
Downstream sink
-> receives schema with chunk + chunk_index
-> key-aware sinks no longer see multiple chunk rows with the same original key only
```
## Merge conclusion
### Conclusion: source blocker cleared from Daniel's side
1. Blocking items
- None from Daniel's source review on this unchanged head.
2. Current gate
- The current CI signal is now green: https://github.com/apache/seatunnel/actions/runs/29234523594/job/86766021047
- Daniel only has `READ` permission on this repository, so GitHub may still show earlier review state and a write-capable maintainer may still need to handle the final approval / dismissal step.
I do not see a reopened code-side blocker from Daniel on the current head.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org