You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "Hisoka-X (via GitHub)" <gi...@apache.org> on 2025/08/18 08:21:37 UTC

[I] Support text splitter transform (Chunking). [seatunnel]

Hisoka-X opened a new issue, #9717:
URL: https://github.com/apache/seatunnel/issues/9717

   (no comment)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Support text splitter transform (Chunking). [seatunnel]

Posted by "iinow (via GitHub)" <gi...@apache.org>.
iinow commented on issue #9717:
URL: https://github.com/apache/seatunnel/issues/9717#issuecomment-3263802300

   @Hisoka-X Additionally, Is it acceptable to extend MultipleFieldOutputTransform, implement protected Column[] getOutputColumns() with only one "chunk" column, and receive the chunk_size option via the constructor for this text chunking transform?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Support text splitter transform (Chunking). [seatunnel]

Posted by "ss666 (via GitHub)" <gi...@apache.org>.
ss666 commented on issue #9717:
URL: https://github.com/apache/seatunnel/issues/9717#issuecomment-4933269652

   Hi, @Hisoka-X and @davidzollo, I'd like to pick this up. 
   
   ### Proposed design
   
   -   Plan to add a `TextChunk` transform under a new `org.apache.seatunnel.transform.chunk` package in `seatunnel-transforms-v2`.
   -   Plan to add these options: `text_field`, `output_field`, `chunk_size`, `overlap_size`, `separators`,following Dify [general mode](https://docs.dify.ai/en/guides/knowledge-base/create-knowledge-and-upload-documents/chunking-and-cleaning-text#general-mode).
   
   ### Question to confirm - Output shape
    
    The earlier discussion suggested extending `MultipleFieldOutputTransform`, but that is a 1-row→1-row map, so a document could only emit **its chunks as a single `ARRAY<STRING>` column**. For a typical RAG pipeline, though, I think the output should be **one chunk per row**, so that each chunk can be embedded and stored as an independent vector record downstream. That means we should instead extend `AbstractCatalogSupportFlatMapTransform` and use `flatMap` (1-row→N-rows) to emit one `STRING` chunk per row.
   
   Once confirmed above, I'll go ahead and open the PR against the latest `dev`. Thanks!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Support text splitter transform (Chunking). [seatunnel]

Posted by "iinow (via GitHub)" <gi...@apache.org>.
iinow commented on issue #9717:
URL: https://github.com/apache/seatunnel/issues/9717#issuecomment-3621269432

   @davidzollo I’m afraid I won’t be able to take on the task as I initially planned. I’m really sorry for the inconvenience


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Support text splitter transform (Chunking). [seatunnel]

Posted by "davidzollo (via GitHub)" <gi...@apache.org>.
davidzollo commented on issue #9717:
URL: https://github.com/apache/seatunnel/issues/9717#issuecomment-3641136040

   > [@davidzollo](https://github.com/davidzollo) I’m afraid I won’t be able to take on the task as I initially planned. I’m really sorry for the inconvenience
   
   Don't worry, Delays do sometimes happen.     Can you complete it some days later?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Support text splitter transform (Chunking). [seatunnel]

Posted by "Hisoka-X (via GitHub)" <gi...@apache.org>.
Hisoka-X commented on issue #9717:
URL: https://github.com/apache/seatunnel/issues/9717#issuecomment-3266827188

   > Is it acceptable to extend MultipleFieldOutputTransform, implement protected Column[] getOutputColumns() 
   
   This is a normal and right way to extend new transform.
   
   > only one "chunk" column, and receive the chunk_size option via the constructor
   
   Could you use https://docs.dify.ai/en/guides/knowledge-base/create-knowledge-and-upload-documents/chunking-and-cleaning-text#general-mode as a guide? We should support general mode in our first version.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Support text splitter transform (Chunking). [seatunnel]

Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on issue #9717:
URL: https://github.com/apache/seatunnel/issues/9717#issuecomment-4934221733

   Thanks for the detailed proposal and for picking this up, @ss666 — and thanks @Hisoka-X for the earlier design guidance.
   
   Your instinct on the output shape is correct. For a RAG pipeline where each chunk needs to be independently embedded and stored as a separate vector record, one-chunk-per-row is the right model. Extending `AbstractCatalogSupportFlatMapTransform` and using `flatMap` (1-row→N-rows) is the better fit here.
   
   The earlier `MultipleFieldOutputTransform` suggestion was reasonable at the time because the initial scope discussion hadn't pinned down the output shape. But as you noted, it is a 1-row→1-row map, so it would only emit a single `ARRAY<STRING>` column — which is not what most RAG downstreams expect.
   
   A few notes on the proposed design:
   
   1. The option set (`text_field`, `output_field`, `chunk_size`, `overlap_size`, `separators`) following Dify general mode looks reasonable for a first version. I'd suggest also thinking about what happens when `separators` is empty or not provided — a fallback chunking strategy (fixed-size with overlap) would be useful.
   
   2. For `flatMap`-based transforms, checkpoint/state behavior is straightforward since the transform is stateless per row, which is good. Just make sure the output `CatalogTable` correctly declares the produced schema (a single `STRING` column for the chunk, plus optionally carrying over source metadata fields if needed).
   
   3. Package name `org.apache.seatunnel.transform.chunk` is fine. The existing transform packages follow a similar flat structure (`split`, `copy`, `jsonpath`, etc.).
   
   4. One thing worth considering: should the transform preserve the original document identifier or index in the output, so downstream stages can correlate chunks back to their source document? This is common in RAG pipelines. It could be an optional `keep_source_field` or similar.
   
   Since @iinow has explicitly stepped away from this issue and there has been no activity for several months, I've removed the old assignment and added the `feature` label. Unfortunately, GitHub does not allow assigning @ss666 to the issue at this time (this typically means the contributor has not yet been added as a collaborator). That's a minor technicality — please go ahead and open the PR, and we can sort out the assignment once it's open.
   
   Looking forward to seeing the PR!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Support text splitter transform (Chunking). [seatunnel]

Posted by "iinow (via GitHub)" <gi...@apache.org>.
iinow commented on issue #9717:
URL: https://github.com/apache/seatunnel/issues/9717#issuecomment-3261508271

   @Hisoka-X 
   
   Can I add a `TextTransform` class to the seatunnel-transforms-v2/src/main/java/org/apache/seatunnel/transforms/text package for this feature?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Support text splitter transform (Chunking). [seatunnel]

Posted by "davidzollo (via GitHub)" <gi...@apache.org>.
davidzollo closed issue #9717: Support text splitter transform (Chunking).
URL: https://github.com/apache/seatunnel/issues/9717


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Support text splitter transform (Chunking). [seatunnel]

Posted by "Hisoka-X (via GitHub)" <gi...@apache.org>.
Hisoka-X commented on issue #9717:
URL: https://github.com/apache/seatunnel/issues/9717#issuecomment-3261656882

   > [@Hisoka-X](https://github.com/Hisoka-X)
   > 
   > Can I add a `TextTransform` class to the seatunnel-transforms-v2/src/main/java/org/apache/seatunnel/transforms/text package for this feature?
   
   That's right!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Support text splitter transform (Chunking). [seatunnel]

Posted by "iinow (via GitHub)" <gi...@apache.org>.
iinow commented on issue #9717:
URL: https://github.com/apache/seatunnel/issues/9717#issuecomment-3289476106

   @Hisoka-X Okay thanks, Can I take this issue?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org