You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "yzeng1618 (via GitHub)" <gi...@apache.org> on 2026/08/10 07:06:51 UTC

[PR] [Improve][Connector-V2] Align Markdown RAG metadata with Knowledge Sync [seatunnel]

yzeng1618 opened a new pull request, #11740:
URL: https://github.com/apache/seatunnel/pull/11740

   ## Purpose
   
   #10990 introduced the Knowledge Sync logical metadata registry and allowed
   `MetadataTransform` to project declared metadata from `SeaTunnelRow.options`.
   The Markdown source still emitted only its physical RAG columns, so a real
   Markdown pipeline could not use the common metadata contract without synthetic
   catalog tables and row options.
   
   This PR adds a connector-local bridge from Markdown RAG output to Knowledge
   Sync logical metadata while preserving all existing physical fields and routing
   formulas.
   
   ## Changes
   
   - Declare `SourceUri`, `DocumentId`, `DocumentHash`, and `ChunkHash` in the
     Markdown source metadata schema when `markdown_rag_metadata_enabled=true`.
   - Populate the four logical keys in every emitted Markdown row's independent
     `SeaTunnelRow.options` map without clearing unrelated options.
   - Build a credential-free logical source identity:
     - preserve the existing local-path normalization for local paths and valid
       `file:` URIs;
     - preserve the scheme, host, explicit port, and path of hierarchical remote
       URIs;
     - remove user info, the complete query component, and the fragment;
     - fail unsafe URI sanitization without echoing the original sensitive value.
   - Build logical `DocumentId` as
     `doc_ + lower_hex(SHA-256(UTF-8(logical SourceUri)))`.
   - Calculate `DocumentHash` from the exact source bytes during the existing
     stream read, before UTF-8 decoding and Markdown parsing, without a second
     source read.
   - Reuse the existing immediate-row text hash as logical `ChunkHash`.
   - Merge registry-defined metadata columns deterministically while preserving
     existing metadata, reusing compatible definitions, and rejecting
     incompatible same-name definitions.
   - Cover bounded existing-file discovery, empty-path fallback, continuous
     discovery, and multi-table catalog construction.
   - Add real Markdown catalog/row coverage to `MetadataTransform`, including
     successful alias projection and existing physical-name collision rejection.
   - Extend the LocalFile Markdown E2E assertion configuration with logical
     metadata projection.
   - Update the English and Chinese File Source and Metadata Transform
     documentation.
   
   ## Compatibility
   
   - No configuration option, dependency, physical column, or public API is added
     or renamed.
   - The behavior remains gated by the existing disabled-by-default
     `markdown_rag_metadata_enabled` option.
   - Existing physical `source_uri`, `document_id`, `chunk_id`, `chunk_index`, and
     `content_hash` fields retain their names, order, values, formulas, and source
     routing behavior.
   - Non-Markdown formats and disabled Markdown RAG configurations retain their
     current metadata-schema behavior.
   - Signed or credential-bearing URLs can intentionally produce different
     physical and logical document IDs: the physical ID remains the compatibility
     value, while the logical ID excludes credentials and transient URI
     components.
   
   ## Semantics and limitations
   
   - The generic bridge removes the complete remote URI query. Resources whose
     identity exists only in query parameters need a stable, non-sensitive path
     or a future source-specific identity mechanism.
   - Logical `ChunkHash` identifies the immediate Markdown output row only. A
     downstream transform that changes or splits text must recompute final
     `ChunkHash`, `ChunkId`, and `ChunkIndex` before a lifecycle sink consumes the
     rows.
   - This bridge does not expose logical `ChunkId` or `ChunkIndex` from the legacy
     physical formulas.
   - Incremental comparison, final-writer affinity, stale-chunk deletion,
     tombstones, and lifecycle sink behavior are outside this PR.
   
   ## Verification
   
   Passed after rebasing onto the latest `apache/seatunnel:dev`:
   
   - Connector-file Markdown reader and catalog tests: 22 tests, 0 failures.
   - `MetadataTransformTest`: 5 tests, 0 failures.
   - Spotless checks executed by both focused Maven test commands.
   - `git diff upstream/dev --check`.
   
   The full root `./mvnw -q -DskipTests verify` reactor was attempted locally but
   did not complete within the available 30-minute execution window. One earlier
   interrupted online run also encountered a transient local Maven resolver
   `Malformed \\uxxxx encoding` error. The full root test suite and LocalFile E2E
   suite were therefore not run locally and should be covered by CI.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Improve][Connector-V2] Align Markdown RAG metadata with Knowledge Sync [seatunnel]

Posted by "dybyte (via GitHub)" <gi...@apache.org>.
dybyte merged PR #11740:
URL: https://github.com/apache/seatunnel/pull/11740


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Improve][Connector-V2] Align Markdown RAG metadata with Knowledge Sync [seatunnel]

Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11740:
URL: https://github.com/apache/seatunnel/pull/11740#issuecomment-5292407878

   Thanks for the update. I checked the current head against the latest `dev` and this branch is currently behind upstream (`behind_by=39`, compare status: `diverged`). The current failing CI signals — `jdbc-connectors-it-part-6 (8, ubuntu-latest)`, `all-connectors-it-7 (8, ubuntu-latest)`, and `unit-test (11, windows-latest)` — do not appear to be directly caused by this PR's diff; `jdbc-connectors-it-part-6` in particular has also failed on `dev`'s own recent CI history, so this looks like pre-existing upstream flakiness rather than something introduced here. The lowest-cost next step would be to sync with the latest `dev` and rerun CI. If it still fails after the sync, happy to help narrow it down against the updated head.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org