You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by li...@apache.org on 2026/01/31 11:17:39 UTC

(seatunnel) branch docs-architecture-design updated (f3368c7244 -> 2be8d7bd0b)

This is an automated email from the ASF dual-hosted git repository.

lidongdai pushed a change to branch docs-architecture-design
in repository https://gitbox.apache.org/repos/asf/seatunnel.git


 discard f3368c7244 docs: align EN architecture docs with Zeta implementation
 discard 7cfab2e2b3 Docs: review and refine architecture documents for accuracy
 discard 20ec27c880 docs(zh): align Jdbc connector name in examples
 discard 378e3be9b2 docs(zh): fix schema evolution config examples
 discard ca3e8c8e52 docs(zh): remove source references and java blocks
 discard 76e8354ec1 Remove tasks/architecture-docs-design.md from PR
 discard 44bd702c17 [Doc] Add comprehensive architecture design documentation
     add 4c110c8fdf [Feature][Connector-V2][HdfsFile] Support true large-file split for parallel read (#10332)
     add 94d6f5c675 [Fix][docs]fix schema-feature.md doc error (#10413)
     add 6948d3492e [Feat][Connector-v2][Kafka] Add support for Schema Registry wire format in Protobuf deserialization (#10183)
     add 70c7040746 [Improve][Common] Improve SinkFlowTestUtils Checkpoint Logic (#10071)
     add 1cfffc7094 [Fix][Connector-V2] Fix NPE when heartbeat.action.query is configured (#10278)
     add 0a4e6ae38e [Fix] [Zeta] Fix Write Count Display Issue in Multi-Sink Scenarios (#10376)
     add 9aea5c9546 [Improve][Zeta] engine-server and seantunnel-ui support remote paginated queries (#9951)
     add 07c0bd1ef3 [Fix][Zeta] Fix memory leak when cancelling pending job (#10315)
     add eca4e2b342 [Improve][Core][Metrics] Refactor and enhance job metrics handling for multiple sinks with updated tests (#10273)
     add 8e27bffff8 [Fix][Connector-v2][MongoDB] There is a problem with using Cache for multi-task submission (#10116)
     add 2be8d7bd0b docs: align EN architecture docs with Zeta implementation

This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version.  This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:

 * -- * -- B -- O -- O -- O   (f3368c7244)
            \
             N -- N -- N   refs/heads/docs-architecture-design (2be8d7bd0b)

You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.

Any revisions marked "omit" are not gone; other references still
refer to them.  Any revisions marked "discard" are gone forever.

No new revisions were added by this update.

Summary of changes:
 .github/workflows/backend.yml                      |   4 +-
 docs/en/connectors/source/HdfsFile.md              |  26 +
 docs/en/connectors/source/Kafka.md                 |  56 +-
 docs/en/connectors/source/LocalFile.md             |  15 +-
 docs/en/engines/zeta/rest-api-v2.md                |  15 +-
 .../introduction/concepts/incompatible-changes.md  |  24 +
 docs/en/introduction/concepts/schema-feature.md    |   2 +-
 docs/sidebars.js                                   |  48 --
 docs/zh/architecture/api-design/catalog-table.md   | 393 --------------
 .../architecture/api-design/sink-architecture.md   | 376 --------------
 .../architecture/api-design/source-architecture.md | 438 ----------------
 docs/zh/architecture/data-flow/multi-table.md      | 434 ----------------
 docs/zh/architecture/design-philosophy.md          | 449 ----------------
 docs/zh/architecture/engine/engine-architecture.md | 574 ---------------------
 docs/zh/architecture/engine/resource-management.md | 539 -------------------
 .../fault-tolerance/checkpoint-mechanism.md        | 549 --------------------
 .../architecture/fault-tolerance/exactly-once.md   | 504 ------------------
 docs/zh/architecture/overview.md                   | 439 ----------------
 .../architecture/translation/translation-layer.md  | 251 ---------
 docs/zh/connectors/source/HdfsFile.md              |  26 +
 docs/zh/connectors/source/Kafka.md                 |  54 ++
 docs/zh/connectors/source/LocalFile.md             |  13 +
 docs/zh/engines/zeta/rest-api-v2.md                |  15 +-
 .../introduction/concepts/incompatible-changes.md  |  24 +
 docs/zh/introduction/concepts/schema-feature.md    |   2 +-
 .../DefaultHeartbeatConnectionProvider.java        |  20 +-
 .../base/relational/JdbcSourceEventDispatcher.java |   3 +
 .../seatunnel/cdc/mongodb/utils/MongodbUtils.java  |  17 +-
 .../reader/fetch/MySqlSourceFetchTaskContext.java  |   8 +
 .../reader/fetch/OracleSourceFetchTaskContext.java |  11 +-
 .../reader/PostgresSourceFetchTaskContext.java     |  14 +
 .../fetch/SqlServerSourceFetchTaskContext.java     |  15 +-
 .../seatunnel/sink/SinkFlowTestUtils.java          | 338 +++++++++++-
 .../file/exception/FileConnectorErrorCode.java     |   5 +-
 .../file/source/BaseMultipleTableFileSource.java   |  24 +-
 .../file/source/reader/AbstractReadStrategy.java   |  22 +-
 .../split/AccordingToSplitSizeSplitStrategy.java   | 242 ++++++---
 .../file/source/split/FileSourceSplit.java         |  14 +
 .../source/split/FileSplitStrategyFactory.java}    |  23 +-
 .../MultipleTableFileSourceSplitEnumerator.java    |  68 ++-
 .../split/MultipleTableFileSplitStrategy.java      |  72 +++
 .../source/split/ParquetFileSplitStrategy.java     |  74 ++-
 .../source/reader/AbstractReadStrategyTest.java    |  81 +++
 .../split/FileSourceSplitCompatibilityTest.java    | 209 ++++++++
 .../seatunnel/file/hdfs/source/HdfsFileSource.java |   6 +-
 .../file/hdfs/source/HdfsFileSourceFactory.java    |  12 +
 ...sFileAccordingToSplitSizeSplitStrategyTest.java | 229 ++++++++
 .../split/HdfsFileSplitStrategyFactoryTest.java    | 123 +++++
 .../file/local/source/LocalFileSource.java         |   9 +-
 ...LocalFileAccordingToSplitSizeSplitStrategy.java |  47 +-
 .../seatunnel/file/local/LocalFileSourceTest.java  |  61 ++-
 .../file/local/SplitFileStrategyTest.java          | 141 ++---
 .../seatunnel/kafka/config/KafkaSourceOptions.java |   9 +
 .../seatunnel/kafka/source/KafkaSourceConfig.java  |  10 +-
 .../seatunnel/kafka/source/KafkaSourceFactory.java |   5 +
 .../src/test/java/mongodb/MongodbCDCIT.java        |  48 ++
 ..._mysql.conf => mongodbcdc_to_mysql_orders.conf} |  11 +-
 .../cdc/mysql/AbstractMysqlCDCITBase.java          |  60 +++
 ....conf => mysqlcdc_to_mysql_with_heartbeat.conf} |   7 +-
 .../seatunnel/cdc/oracle/OracleCDCIT.java          |  34 ++
 ...onf => oraclecdc_to_oracle_with_heartbeat.conf} |   5 +-
 .../seatunnel/cdc/postgres/PostgresCDCIT.java      |  60 +++
 ...=> postgrescdc_to_postgres_with_heartbeat.conf} |   4 +
 .../connector/cdc/sqlserver/SqlServerCDCIT.java    | 163 ++++--
 ...=> sqlservercdc_to_console_with_heartbeat.conf} |   4 +
 .../e2e/connector/file/hdfs/HdfsFileIT.java        |  41 ++
 ...sert.conf => hdfs_parquet_split_to_assert.conf} |   3 +-
 ..._assert.conf => hdfs_text_split_to_assert.conf} |  16 +-
 .../seatunnel/e2e/connector/kafka/KafkaIT.java     | 100 ++++
 ...chema_registry_header_transform_to_assert.conf} |  10 +-
 .../connector-mongodb-e2e/pom.xml                  |   7 +
 .../e2e/connector/v2/mongodb/MongodbIT.java        | 103 +++-
 .../fake_source_to_transaction_sink_mongodb.conf   |  74 ---
 .../seatunnel/engine/e2e/CommittedMetricsIT.java   |  17 +-
 .../seatunnel/engine/e2e/MultiTableMetricsIT.java  |  32 +-
 .../org/apache/seatunnel/engine/e2e/RestApiIT.java |   8 +-
 .../seatunnel/engine/client/job/JobClient.java     |  25 +-
 .../seatunnel/engine/client/JobClientTest.java     |  19 +-
 .../engine/core/parse/JobConfigParser.java         |   6 +-
 .../engine/server/CoordinatorService.java          |   7 -
 .../engine/server/dag/physical/PhysicalPlan.java   |   1 +
 .../engine/server/rest/service/BaseService.java    | 332 +++++++++++-
 .../engine/server/rest/service/JobInfoService.java |   4 +
 .../server/rest/servlet/FinishedJobsServlet.java   |   4 +-
 .../server/rest/servlet/PageBaseServlet.java       |  75 +++
 .../server/rest/servlet/RunningJobsServlet.java    |   4 +-
 ...CoordinatorServiceWithCancelPendingJobTest.java |  20 +-
 .../engine/server/rest/RestApiHttpsTest.java       | 273 +++++++++-
 .../server/rest/RestApiRequestCallback.java}       |   9 +-
 .../rest/service/BaseServiceTableMetricsTest.java  | 288 ++++++++++-
 .../seatunnel-engine-ui/src/service/job/index.ts   |   6 +-
 .../seatunnel-engine-ui/src/service/job/types.ts   |   5 +
 .../seatunnel-engine-ui/src/tests/jobs.spec.ts     |  26 +-
 .../src/views/jobs/finished-jobs.tsx               |  25 +-
 .../src/views/jobs/running-jobs.tsx                |  25 +-
 .../protobuf/ProtobufDeserializationSchema.java    |  15 +-
 ...RegistryAwareProtobufDeserializationSchema.java | 129 +++++
 ...stryAwareProtobufDeserializationSchemaTest.java | 192 +++++++
 tasks/docs-zh-architecture-accuracy-review.md      | 187 -------
 99 files changed, 3910 insertions(+), 5756 deletions(-)
 delete mode 100644 docs/zh/architecture/api-design/catalog-table.md
 delete mode 100644 docs/zh/architecture/api-design/sink-architecture.md
 delete mode 100644 docs/zh/architecture/api-design/source-architecture.md
 delete mode 100644 docs/zh/architecture/data-flow/multi-table.md
 delete mode 100644 docs/zh/architecture/design-philosophy.md
 delete mode 100644 docs/zh/architecture/engine/engine-architecture.md
 delete mode 100644 docs/zh/architecture/engine/resource-management.md
 delete mode 100644 docs/zh/architecture/fault-tolerance/checkpoint-mechanism.md
 delete mode 100644 docs/zh/architecture/fault-tolerance/exactly-once.md
 delete mode 100644 docs/zh/architecture/overview.md
 delete mode 100644 docs/zh/architecture/translation/translation-layer.md
 copy seatunnel-api/src/main/java/org/apache/seatunnel/api/table/catalog/SQLPreviewResult.java => seatunnel-connectors-v2/connector-cdc/connector-cdc-base/src/main/java/io/debezium/heartbeat/DefaultHeartbeatConnectionProvider.java (68%)
 rename seatunnel-connectors-v2/connector-file/{connector-file-local/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/local/source/split/LocalFileSplitStrategyFactory.java => connector-file-base/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/source/split/FileSplitStrategyFactory.java} (79%)
 create mode 100644 seatunnel-connectors-v2/connector-file/connector-file-base/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/source/split/MultipleTableFileSplitStrategy.java
 create mode 100644 seatunnel-connectors-v2/connector-file/connector-file-base/src/test/java/org/apache/seatunnel/connectors/seatunnel/file/source/split/FileSourceSplitCompatibilityTest.java
 create mode 100644 seatunnel-connectors-v2/connector-file/connector-file-hadoop/src/test/java/org/apache/seatunnel/connectors/seatunnel/file/hdfs/source/split/HdfsFileAccordingToSplitSizeSplitStrategyTest.java
 create mode 100644 seatunnel-connectors-v2/connector-file/connector-file-hadoop/src/test/java/org/apache/seatunnel/connectors/seatunnel/file/hdfs/source/split/HdfsFileSplitStrategyFactoryTest.java
 copy seatunnel-e2e/seatunnel-connector-v2-e2e/connector-cdc-mongodb-e2e/src/test/resources/{mongodbcdc_to_mysql.conf => mongodbcdc_to_mysql_orders.conf} (89%)
 copy seatunnel-e2e/seatunnel-connector-v2-e2e/connector-cdc-mysql-e2e/src/test/resources/{mysqlcdc_to_mysql.conf => mysqlcdc_to_mysql_with_heartbeat.conf} (93%)
 copy seatunnel-e2e/seatunnel-connector-v2-e2e/connector-cdc-oracle-e2e/src/test/resources/{oraclecdc_to_oracle_use_select_count.conf => oraclecdc_to_oracle_with_heartbeat.conf} (93%)
 copy seatunnel-e2e/seatunnel-connector-v2-e2e/connector-cdc-postgres-e2e/src/test/resources/{postgrescdc_to_postgres.conf => postgrescdc_to_postgres_with_heartbeat.conf} (92%)
 copy seatunnel-e2e/seatunnel-connector-v2-e2e/connector-cdc-sqlserver-e2e/src/test/resources/{sqlservercdc_to_console.conf => sqlservercdc_to_console_with_heartbeat.conf} (91%)
 copy seatunnel-e2e/seatunnel-connector-v2-e2e/connector-file-hadoop-e2e/src/test/resources/{hdfs_normal_to_assert.conf => hdfs_parquet_split_to_assert.conf} (97%)
 copy seatunnel-e2e/seatunnel-connector-v2-e2e/connector-file-hadoop-e2e/src/test/resources/{hdfs_empty_text_to_assert.conf => hdfs_text_split_to_assert.conf} (73%)
 copy seatunnel-e2e/seatunnel-connector-v2-e2e/connector-kafka-e2e/src/test/resources/protobuf/{kafka_protobuf_transform_to_assert.conf => kafka_protobuf_schema_registry_header_transform_to_assert.conf} (94%)
 delete mode 100644 seatunnel-e2e/seatunnel-connector-v2-e2e/connector-mongodb-e2e/src/test/resources/transactionIT/fake_source_to_transaction_sink_mongodb.conf
 create mode 100644 seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/rest/servlet/PageBaseServlet.java
 copy seatunnel-engine/seatunnel-engine-server/src/{main/java/org/apache/seatunnel/engine/server/rest/ErrResponse.java => test/java/org/apache/seatunnel/engine/server/rest/RestApiRequestCallback.java} (87%)
 create mode 100644 seatunnel-formats/seatunnel-format-protobuf/src/main/java/org/apache/seatunnel/format/protobuf/SchemaRegistryAwareProtobufDeserializationSchema.java
 create mode 100644 seatunnel-formats/seatunnel-format-protobuf/src/test/java/org/apache/seatunnel/format/protobuf/SchemaRegistryAwareProtobufDeserializationSchemaTest.java
 delete mode 100644 tasks/docs-zh-architecture-accuracy-review.md