You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "JNSimba (via GitHub)" <gi...@apache.org> on 2026/04/15 11:41:11 UTC
[PR] [Improve](streaming-job) use per-table publication instead of ALL TABLES for PostgreSQL CDC [doris]
JNSimba opened a new pull request, #62526:
URL: https://github.com/apache/doris/pull/62526
### What problem does this PR solve?
Problem Summary:
Previously, PostgreSQL CDC streaming jobs created publications using `FOR ALL TABLES`, which monitors all tables in the database regardless of which tables the job actually needs to capture. This causes unnecessary WAL decoding overhead and network traffic. Additionally, users could not specify custom `slot_name` or `publication_name`, and these values were not visible in `SHOW JOBS`.
This PR makes the following improvements:
1. **Per-table publication (FILTERED mode)**: Publication is now created with `FOR TABLE table1, table2` instead of `FOR ALL TABLES`, only including tables specified in `include_tables`.
2. **Configurable slot_name and publication_name**: Users can optionally pass `slot_name` and `publication_name` in source properties. If not specified, auto-generated names with format `doris_cdc_{jobId}_{uuid8}` / `doris_pub_{jobId}_{uuid8}` are used.
3. **Visible in SHOW**: Auto-generated or user-provided slot/publication names are persisted in `sourceProperties` and visible via `SHOW JOBS` / TVF query.
4. **Partition table support for FILTERED mode**: Added `publish_via_partition_root = true` option for PostgreSQL 13+ in FILTERED mode (previously only in ALL_TABLES mode).
5. **Proper cleanup**: Auto-created publications and slots (with `doris_pub_`/`doris_cdc_` prefix) are dropped when the job is deleted; user-provided ones are preserved.
### Release note
Support per-table PostgreSQL publication for streaming CDC jobs. Users can now optionally specify `slot_name` and `publication_name` in source properties. Auto-generated slot/publication names are visible in SHOW JOBS output.
### Check List (For Author)
- Test
- [x] Regression test
- [x] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason
- Behavior changed:
- [x] Yes. PostgreSQL CDC publication changed from `FOR ALL TABLES` to `FOR TABLE` (filtered mode). Slot and publication names now use `{jobId}_{uuid8}` suffix instead of just `{jobId}`. Both are now visible in SHOW JOBS output.
- Does this need documentation?
- [x] Yes. <!-- Documentation for new slot_name/publication_name source properties -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improve](streaming-job) use per-table publication instead of ALL TABLES for PostgreSQL CDC [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62526:
URL: https://github.com/apache/doris/pull/62526#issuecomment-4257914838
# FE Regression Coverage Report
Increment line coverage ` 0.00% (0/7)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62526_3fcc978b693bf9d873529667f3314d29372459c5_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62526_3fcc978b693bf9d873529667f3314d29372459c5_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improve](streaming-job) use per-table publication instead of ALL TABLES for PostgreSQL CDC [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62526:
URL: https://github.com/apache/doris/pull/62526#issuecomment-4268131149
# FE UT Coverage Report
Increment line coverage ` 4.27% (5/117)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62526_8c54460b9b6a624fa144b975e75eb850fa17265e/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62526_8c54460b9b6a624fa144b975e75eb850fa17265e/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improve](streaming-job) use per-table publication instead of ALL TABLES for PostgreSQL CDC [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #62526:
URL: https://github.com/apache/doris/pull/62526#discussion_r3098593113
##########
fs_brokers/cdc_client/src/main/java/org/apache/doris/cdcclient/source/reader/postgres/PostgresSourceReader.java:
##########
@@ -255,8 +273,26 @@ private PostgresSourceConfig generatePostgresConfig(
return configFactory.create(subtaskId);
}
- private String getSlotName(String jobId) {
- return "doris_cdc_" + jobId;
+ private String resolveSlotName(Map<String, String> config, String jobId) {
+ String name = config.get(DataSourceConfigKeys.SLOT_NAME);
+ return StringUtils.isNotBlank(name) ? name : DataSourceConfigKeys.defaultSlotName(jobId);
+ }
+
+ private String resolvePublicationName(Map<String, String> config, String jobId) {
+ String name = config.get(DataSourceConfigKeys.PUBLICATION_NAME);
+ return StringUtils.isNotBlank(name)
+ ? name
+ : DataSourceConfigKeys.defaultPublicationName(jobId);
Review Comment:
Changing the fallback publication name here breaks restart and rolling-upgrade compatibility for existing PostgreSQL streaming jobs. Older jobs persisted no `publication_name`, and their effective publication was the Debezium default `dbz_publication`. After FE/BE restart on this version, the same persisted job will now reconnect as `doris_pub_{jobId}`, which can fail without CREATE/ownership privileges and also leaves the old publication orphaned. We need a compatibility fallback for pre-existing jobs before switching the default.
##########
fe/fe-core/src/main/java/org/apache/doris/job/extensions/insert/streaming/PostgresResourceValidator.java:
##########
@@ -0,0 +1,172 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.job.extensions.insert.streaming;
+
+import org.apache.doris.datasource.jdbc.client.JdbcClient;
+import org.apache.doris.job.cdc.DataSourceConfigKeys;
+import org.apache.doris.job.common.DataSourceType;
+import org.apache.doris.job.exception.JobException;
+import org.apache.doris.job.util.StreamingJobUtils;
+
+import org.apache.commons.lang3.StringUtils;
+
+import java.sql.Connection;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ * Fail-fast validation of PostgreSQL replication slot and publication at CREATE JOB time,
+ * before the CDC client connects. Catches mistakes (user-provided slot/publication missing,
+ * conflicting jobId) with actionable errors. Validation runs only on initial create; restarts
+ * skip this path by design, so an active slot held by the previous BE does not self-conflict.
+ */
+public class PostgresResourceValidator {
+
+ public static void validate(Map<String, String> sourceProperties, String jobId, List<String> tableNames)
+ throws JobException {
+ String slotName = resolveSlotName(sourceProperties, jobId);
+ String publicationName = resolvePublicationName(sourceProperties, jobId);
+ boolean slotUserProvided = isSlotUserProvided(sourceProperties);
+ boolean pubUserProvided = isPublicationUserProvided(sourceProperties);
+ String pgSchema = sourceProperties.get(DataSourceConfigKeys.SCHEMA);
+ List<String> qualifiedTables = new ArrayList<>();
+ for (String name : tableNames) {
+ qualifiedTables.add(pgSchema + "." + name);
+ }
+
+ JdbcClient jdbcClient = StreamingJobUtils.getJdbcClient(DataSourceType.POSTGRES, sourceProperties);
Review Comment:
This new validation path allocates a `JdbcClient`/HikariDataSource but never calls `closeClient()`. Only the JDBC `Connection` is closed here. Since `validate()` now runs for every CREATE JOB and PG TVF validation, FE will leak datasource/classloader resources on each call. Please release the `JdbcClient` in a `finally` block, like other JdbcClient call sites do.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improve](streaming-job) use per-table publication instead of ALL TABLES for PostgreSQL CDC [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #62526:
URL: https://github.com/apache/doris/pull/62526#issuecomment-4265132328
OpenCode automated review failed and did not complete.
Error: Review step was skipped (possibly timeout or cancelled)
Workflow run: https://github.com/apache/doris/actions/runs/24546598676
Please inspect the workflow logs and rerun the review after the underlying issue is resolved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improve](streaming-job) use per-table publication instead of ALL TABLES for PostgreSQL CDC [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #62526:
URL: https://github.com/apache/doris/pull/62526#discussion_r3098593113
##########
fs_brokers/cdc_client/src/main/java/org/apache/doris/cdcclient/source/reader/postgres/PostgresSourceReader.java:
##########
@@ -255,8 +273,26 @@ private PostgresSourceConfig generatePostgresConfig(
return configFactory.create(subtaskId);
}
- private String getSlotName(String jobId) {
- return "doris_cdc_" + jobId;
+ private String resolveSlotName(Map<String, String> config, String jobId) {
+ String name = config.get(DataSourceConfigKeys.SLOT_NAME);
+ return StringUtils.isNotBlank(name) ? name : DataSourceConfigKeys.defaultSlotName(jobId);
+ }
+
+ private String resolvePublicationName(Map<String, String> config, String jobId) {
+ String name = config.get(DataSourceConfigKeys.PUBLICATION_NAME);
+ return StringUtils.isNotBlank(name)
+ ? name
+ : DataSourceConfigKeys.defaultPublicationName(jobId);
Review Comment:
Changing the fallback publication name here breaks restart and rolling-upgrade compatibility for existing PostgreSQL streaming jobs. Older jobs persisted no , and their effective publication was Debezium's default . After FE/BE restart on this version, the same persisted job will now reconnect as , which can fail without CREATE/ownership privileges and also leaves the old publication orphaned. We need a compatibility fallback for pre-existing jobs before switching the default.
##########
fe/fe-core/src/main/java/org/apache/doris/job/extensions/insert/streaming/PostgresResourceValidator.java:
##########
@@ -0,0 +1,172 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.job.extensions.insert.streaming;
+
+import org.apache.doris.datasource.jdbc.client.JdbcClient;
+import org.apache.doris.job.cdc.DataSourceConfigKeys;
+import org.apache.doris.job.common.DataSourceType;
+import org.apache.doris.job.exception.JobException;
+import org.apache.doris.job.util.StreamingJobUtils;
+
+import org.apache.commons.lang3.StringUtils;
+
+import java.sql.Connection;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ * Fail-fast validation of PostgreSQL replication slot and publication at CREATE JOB time,
+ * before the CDC client connects. Catches mistakes (user-provided slot/publication missing,
+ * conflicting jobId) with actionable errors. Validation runs only on initial create; restarts
+ * skip this path by design, so an active slot held by the previous BE does not self-conflict.
+ */
+public class PostgresResourceValidator {
+
+ public static void validate(Map<String, String> sourceProperties, String jobId, List<String> tableNames)
+ throws JobException {
+ String slotName = resolveSlotName(sourceProperties, jobId);
+ String publicationName = resolvePublicationName(sourceProperties, jobId);
+ boolean slotUserProvided = isSlotUserProvided(sourceProperties);
+ boolean pubUserProvided = isPublicationUserProvided(sourceProperties);
+ String pgSchema = sourceProperties.get(DataSourceConfigKeys.SCHEMA);
+ List<String> qualifiedTables = new ArrayList<>();
+ for (String name : tableNames) {
+ qualifiedTables.add(pgSchema + "." + name);
+ }
+
+ JdbcClient jdbcClient = StreamingJobUtils.getJdbcClient(DataSourceType.POSTGRES, sourceProperties);
Review Comment:
This new validation path allocates a /HikariDataSource but never calls . Only the JDBC is closed here. Since now runs for every CREATE JOB and PG TVF validation, FE will leak datasource/classloader resources on each call. Please release the in a block, like the other JdbcClient call sites do.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improve](streaming-job) use per-table publication instead of ALL TABLES for PostgreSQL CDC [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62526:
URL: https://github.com/apache/doris/pull/62526#issuecomment-4279112435
# FE Regression Coverage Report
Increment line coverage ` 1.35% (2/148)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62526_8730818d26e7573135450b76ea7eb1b66dd3b0b0_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62526_8730818d26e7573135450b76ea7eb1b66dd3b0b0_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improve](streaming-job) use per-table publication instead of ALL TABLES for PostgreSQL CDC [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #62526:
URL: https://github.com/apache/doris/pull/62526#discussion_r3108084151
##########
fe/fe-core/src/main/java/org/apache/doris/tablefunction/CdcStreamTableValuedFunction.java:
##########
@@ -61,23 +61,32 @@ public CdcStreamTableValuedFunction(Map<String, String> properties) throws Analy
private void processProps(Map<String, String> properties) throws AnalysisException {
Map<String, String> copyProps = new HashMap<>(properties);
copyProps.put("format", "json");
+
+ // Standalone TVF: random jobId. TVF-in-job: job.id injected by rewriteTvfParams.
+ String jobId = copyProps.computeIfAbsent(JOB_ID_KEY,
+ k -> UUID.randomUUID().toString().replace("-", ""));
+
+ // Default PG slot/pub so cdcclient auto-creates per-job resources
+ StreamingJobUtils.populateDefaultSourceProperties(
Review Comment:
Blocking: this changes the effective publication name for legacy `cdc_stream` streaming jobs after FE restart/upgrade.
For jobs created before this PR, the persisted SQL does not contain `publication_name`, so the old TVF path used Debezium's default `dbz_publication`. After replay, `StreamingInsertJob` rebuilds the TVF from that old SQL, comes through `processProps()`, and this line now injects `doris_pub_{jobId}` instead. The FROM-TO path has a compatibility fallback in `PostgresSourceReader.resolvePublicationName()`, but the TVF path never reaches it because the new name is materialized here first.
That means an existing TVF-backed PG job can reconnect with a different publication after upgrade, require new CREATE/ownership privileges, and skip cleaning up the legacy publication it previously used. We need a compatibility branch for replayed jobs whose SQL/persisted props do not already carry `publication_name`.
##########
fe/fe-core/src/main/java/org/apache/doris/job/extensions/insert/streaming/PostgresResourceValidator.java:
##########
@@ -0,0 +1,169 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.job.extensions.insert.streaming;
+
+import org.apache.doris.datasource.jdbc.client.JdbcClient;
+import org.apache.doris.job.cdc.DataSourceConfigKeys;
+import org.apache.doris.job.common.DataSourceType;
+import org.apache.doris.job.exception.JobException;
+import org.apache.doris.job.util.StreamingJobUtils;
+
+import org.apache.commons.lang3.StringUtils;
+
+import java.sql.Connection;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ * Fail-fast validation of PostgreSQL replication slot and publication at CREATE JOB time,
+ * before the CDC client connects. Catches mistakes (user-provided slot/publication missing,
+ * conflicting jobId) with actionable errors. Validation runs only on initial create; restarts
+ * skip this path by design, so an active slot held by the previous BE does not self-conflict.
+ */
+public class PostgresResourceValidator {
+
+ public static void validate(Map<String, String> sourceProperties, String jobId, List<String> tableNames)
+ throws JobException {
+ String slotName = resolveSlotName(sourceProperties, jobId);
+ String publicationName = resolvePublicationName(sourceProperties, jobId);
+ // Pattern-match ownership: name equals the default = Doris-owned (auto); otherwise user.
+ String defaultSlot = DataSourceConfigKeys.defaultSlotName(jobId);
+ String defaultPub = DataSourceConfigKeys.defaultPublicationName(jobId);
+ boolean slotUserProvided = !defaultSlot.equals(slotName);
+ boolean pubUserProvided = !defaultPub.equals(publicationName);
+ String pgSchema = sourceProperties.get(DataSourceConfigKeys.SCHEMA);
+ List<String> qualifiedTables = new ArrayList<>();
+ for (String name : tableNames) {
+ qualifiedTables.add(pgSchema + "." + name);
+ }
+
+ JdbcClient jdbcClient = StreamingJobUtils.getJdbcClient(DataSourceType.POSTGRES, sourceProperties);
+ try (Connection conn = jdbcClient.getConnection()) {
+ boolean pubExists = publicationExists(conn, publicationName);
+ if (!pubExists && pubUserProvided) {
+ throw new JobException(
+ "publication does not exist: " + publicationName
+ + ". Create it before starting the job or omit "
+ + DataSourceConfigKeys.PUBLICATION_NAME
+ + " to let Doris create one.");
+ }
+ if (pubExists) {
+ List<String> missing = findMissingTables(conn, publicationName, qualifiedTables);
Review Comment:
Blocking: for partitioned root tables, "publication exists and covers the table name" is not a sufficient correctness check.
A user-managed publication created as `CREATE PUBLICATION pub FOR TABLE schema.orders` passes `pg_publication_tables`, but if `pg_publication.pubviaroot` is false (the default) PostgreSQL publishes changes under the child partition identities. Doris still builds its capture set from `include_tables=orders`, so those partition events can be silently skipped even though validation succeeded.
The auto-created path avoids this by using `WITH (publish_via_partition_root = true)`, so the validator needs to reject or repair existing publications that do not have `pubviaroot` when any configured table is a partitioned root.
##########
fs_brokers/cdc_client/src/main/java/org/apache/doris/cdcclient/source/reader/postgres/PostgresSourceReader.java:
##########
@@ -93,9 +95,13 @@ public PostgresSourceReader() {
public void initialize(String jobId, DataSource dataSource, Map<String, String> config) {
PostgresSourceConfig sourceConfig = generatePostgresConfig(config, jobId, 0);
PostgresDialect dialect = new PostgresDialect(sourceConfig);
- synchronized (SLOT_CREATION_LOCK) {
- LOG.info("Creating slot for job {}, user {}", jobId, sourceConfig.getUsername());
- createSlotForGlobalStreamSplit(dialect);
+ // Only create the slot when Doris owns it (name == default); user-provided slots must
+ // pre-exist, validated at CREATE JOB.
Review Comment:
Blocking: this only skips the eager FE-side slot creation. It does not stop Debezium from auto-creating a missing user-supplied slot later.
`PostgresReplicationConnection.initReplicationSlot()` still calls `createReplicationSlot()` whenever `getSlotInfo()` returns `INVALID`. So on the standalone TVF path, `select * from cdc_stream(..., "slot_name" = "foo")` will create `foo` if it does not already exist, and `close()` will preserve it because `isSlotDorisOwned()` is false. That leaks a replication slot for every typo/missing user slot instead of failing.
If the ownership contract is "user-provided slot must pre-exist and Doris never owns/drops it", the cdc client also has to reject missing user-owned slots rather than auto-creating them.
##########
fe/fe-core/src/main/java/org/apache/doris/job/util/StreamingJobUtils.java:
##########
@@ -448,6 +449,73 @@ public static List<Column> getColumns(JdbcClient jdbcClient,
* The remoteDB implementation differs for each data source;
* refer to the hierarchical mapping in the JDBC catalog.
*/
+ /**
+ * Populate default resource names into properties, then validate. No-op for sources that
+ * don't need it. Mutates properties: callers should expect default values to be inserted.
+ */
+ public static void resolveAndValidateSource(DataSourceType sourceType,
+ Map<String, String> properties,
+ String jobId,
+ List<String> tables) throws JobException {
+ if (sourceType == DataSourceType.POSTGRES) {
+ // PG slot/pub: values equal to default = Doris-owned; any other value = user-owned.
+ // (users cannot specify default names since jobId is unknown pre-CREATE)
+ properties.putIfAbsent(DataSourceConfigKeys.SLOT_NAME,
+ DataSourceConfigKeys.defaultSlotName(jobId));
+ properties.putIfAbsent(DataSourceConfigKeys.PUBLICATION_NAME,
+ DataSourceConfigKeys.defaultPublicationName(jobId));
+ validateSource(sourceType, properties, jobId, tables);
+ }
+ }
+
+ public static void validateSource(DataSourceType sourceType,
+ Map<String, String> properties,
+ String jobId,
+ List<String> tables) throws JobException {
+ if (sourceType == DataSourceType.POSTGRES) {
+ PostgresResourceValidator.validate(properties, jobId, tables);
+ }
+ }
+
+ /**
+ * Validate source-side resources for a streaming job backed by a TVF. Only cdc_stream
+ * TVF is subject to source validation (e.g. PG slot/publication ownership); other TVFs
+ * (s3, ...) are no-ops.
+ *
+ * <p>originTvfProps is treated as read-only (Nereids may hand back an immutable map).
+ * Defaults are populated into a temporary copy so ownership checks see the effective
+ * slot/pub names without mutating the caller's map.
+ */
+ public static void validateTvfSource(String tvfType,
+ Map<String, String> originTvfProps,
+ String jobId) throws JobException {
+ if (!"cdc_stream".equalsIgnoreCase(tvfType)) {
+ return;
+ }
+ DataSourceType sourceType = DataSourceType.valueOf(
+ originTvfProps.get(DataSourceConfigKeys.TYPE).toUpperCase());
+ List<String> tables = Collections.singletonList(
+ originTvfProps.get(DataSourceConfigKeys.TABLE));
+ Map<String, String> effective = new HashMap<>(originTvfProps);
+ populateDefaultSourceProperties(sourceType, effective, jobId);
Review Comment:
Blocking: the `CREATE JOB ... DO INSERT ... SELECT FROM cdc_stream(...)` path still bypasses the new FE identifier validation.
`CreateJobCommand.validate()` only calls `DataSourceConfigValidator.validateSource()` for `FROM POSTGRES` jobs. Here `validateTvfSource()` goes straight to `PostgresResourceValidator`, so `slot_name` / `publication_name` values like `Bad-Name` or `pub;drop` are accepted in the TVF-backed job path even though the PR description says CREATE-time validation now covers both paths. Those raw values later reach the cdc client DDL and fail only at runtime.
Please run the same `DataSourceConfigValidator.validateSource()` checks on the TVF props before `PostgresResourceValidator.validate()`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improve](streaming-job) use per-table publication instead of ALL TABLES for PostgreSQL CDC [doris]
Posted by "JNSimba (via GitHub)" <gi...@apache.org>.
JNSimba commented on PR #62526:
URL: https://github.com/apache/doris/pull/62526#issuecomment-4277517496
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improve](streaming-job) use per-table publication instead of ALL TABLES for PostgreSQL CDC [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #62526:
URL: https://github.com/apache/doris/pull/62526#discussion_r3097642667
##########
fs_brokers/cdc_client/src/main/java/org/apache/doris/cdcclient/source/reader/postgres/PostgresSourceReader.java:
##########
@@ -53,16 +53,23 @@
import org.apache.flink.cdc.connectors.postgres.source.utils.TableDiscoveryUtils;
import org.apache.flink.table.types.DataType;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
import java.time.Duration;
import java.time.Instant;
+import java.util.ArrayList;
import java.util.Collections;
import java.util.HashMap;
+import java.util.HashSet;
import java.util.List;
import java.util.Map;
import java.util.Properties;
+import java.util.Set;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.google.common.base.Preconditions;
+import io.debezium.connector.postgresql.PostgresConnectorConfig;
+import io.debezium.connector.postgresql.PostgresConnectorConfig.AutoCreateMode;
Review Comment:
`slot_name` is treated as "user-owned" here, but the downstream Postgres replication path can still create that slot later if it doesn't already exist. `initialize()` skips `createSlotForGlobalStreamSplit()` when `slot_name` is set, yet `PostgresReplicationConnection.initReplicationSlot()` still calls `createReplicationSlot()` whenever `getSlotInfo()` returns `INVALID`. That means `CREATE JOB ... "slot_name" = "my_slot"` will silently create `my_slot` if the caller forgot to pre-create it, and `close()` will then preserve that Doris-created slot forever because `isSlotUserProvided(config)` is true. Can we fail fast here by verifying that a user-provided slot actually exists (and probably that it is compatible) before we mark it as hands-off?
##########
fs_brokers/cdc_client/src/main/java/org/apache/doris/cdcclient/source/reader/postgres/PostgresSourceReader.java:
##########
@@ -243,7 +275,7 @@ private PostgresSourceConfig generatePostgresConfig(
Review Comment:
This existence/table-coverage check is not sufficient for the partition-table support added by this PR. If a user supplies an existing publication for a partitioned root table but that publication was created without `publish_via_partition_root = true`, PostgreSQL will publish changes using the individual partition identities. Doris still builds `table.include.list` from the root names in `include_tables`, so those child-table events won't match the configured capture set. In other words, a manual/legacy publication can pass validation here and still silently miss partition changes. We should also validate (or repair for Doris-owned publications) `pg_publication.pubviaroot` when any configured table is a partitioned root.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improve](streaming-job) use per-table publication instead of ALL TABLES for PostgreSQL CDC [doris]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #62526:
URL: https://github.com/apache/doris/pull/62526#discussion_r3086255453
##########
fs_brokers/cdc_client/src/main/java/org/apache/doris/cdcclient/source/reader/postgres/PostgresSourceReader.java:
##########
@@ -227,6 +227,13 @@ private PostgresSourceConfig generatePostgresConfig(
Properties dbzProps = ConfigUtil.getDefaultDebeziumProps();
dbzProps.put("interval.handling.mode", "string");
+
+ // Configure publication: use filtered mode to create per-table publication
+ String publicationName =
+ cdcConfig.getOrDefault(DataSourceConfigKeys.PUBLICATION_NAME, "doris_pub_" + jobId);
Review Comment:
The default publication name fallback is hard-coded as `"doris_pub_" + jobId` instead of using the shared constants/helpers in `DataSourceConfigKeys` (e.g. `DEFAULT_PUBLICATION_PREFIX`). This duplicates naming logic across modules and risks divergence from FE defaults.
```suggestion
String publicationName = cdcConfig.getOrDefault(
DataSourceConfigKeys.PUBLICATION_NAME,
DataSourceConfigKeys.DEFAULT_PUBLICATION_PREFIX + jobId);
```
##########
regression-test/suites/job_p0/streaming_job/cdc/test_streaming_postgres_job_publication.groovy:
##########
@@ -0,0 +1,166 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+
+import org.awaitility.Awaitility
+
+import static java.util.concurrent.TimeUnit.SECONDS
+
+suite("test_streaming_postgres_job_publication", "p0,external,pg,external_docker,external_docker_pg,nondatalake") {
+ def jobName = "test_pg_pub_job"
+ def currentDb = (sql "select database()")[0][0]
+ def table1 = "pub_test_table1"
+ def table2 = "pub_test_table2"
+ def pgDB = "postgres"
+ def pgSchema = "cdc_test"
+ def pgUser = "postgres"
+ def pgPassword = "123456"
+
+ sql """DROP JOB IF EXISTS where jobname = '${jobName}'"""
+ sql """drop table if exists ${currentDb}.${table1} force"""
+ sql """drop table if exists ${currentDb}.${table2} force"""
+
+ String enabled = context.config.otherConfigs.get("enableJdbcTest")
+ if (enabled != null && enabled.equalsIgnoreCase("true")) {
+ String pg_port = context.config.otherConfigs.get("pg_14_port");
+ String externalEnvIp = context.config.otherConfigs.get("externalEnvIp")
+ String s3_endpoint = getS3Endpoint()
+ String bucket = getS3BucketName()
+ String driver_url = "https://${bucket}.${s3_endpoint}/regression/jdbc_driver/postgresql-42.5.0.jar"
+
+ // Setup PG tables
+ connect("${pgUser}", "${pgPassword}", "jdbc:postgresql://${externalEnvIp}:${pg_port}/${pgDB}") {
+ sql """DROP TABLE IF EXISTS ${pgDB}.${pgSchema}.${table1}"""
+ sql """DROP TABLE IF EXISTS ${pgDB}.${pgSchema}.${table2}"""
+ sql """CREATE TABLE ${pgDB}.${pgSchema}.${table1} (
+ "id" int PRIMARY KEY,
+ "name" varchar(200)
+ )"""
+ sql """CREATE TABLE ${pgDB}.${pgSchema}.${table2} (
+ "id" int PRIMARY KEY,
+ "name" varchar(200)
+ )"""
+ sql """INSERT INTO ${pgDB}.${pgSchema}.${table1} (id, name) VALUES (1, 'Alice');"""
+ sql """INSERT INTO ${pgDB}.${pgSchema}.${table2} (id, name) VALUES (1, 'Bob');"""
+ }
+
+ // ========== Test 1: auto-generated slot_name and publication_name ==========
+ sql """CREATE JOB ${jobName}
+ ON STREAMING
+ FROM POSTGRES (
+ "jdbc_url" = "jdbc:postgresql://${externalEnvIp}:${pg_port}/${pgDB}",
+ "driver_url" = "${driver_url}",
+ "driver_class" = "org.postgresql.Driver",
+ "user" = "${pgUser}",
+ "password" = "${pgPassword}",
+ "database" = "${pgDB}",
+ "schema" = "${pgSchema}",
+ "include_tables" = "${table1},${table2}",
+ "offset" = "initial"
+ )
+ TO DATABASE ${currentDb} (
+ "table.create.properties.replication_num" = "1"
+ )
+ """
+
+ // Verify SHOW contains auto-generated slot_name and publication_name
+ def jobInfo = sql """select ExecuteSql from jobs("type"="insert") where Name='${jobName}'"""
+ log.info("jobInfo ExecuteSql: " + jobInfo)
+ def executeSql = jobInfo.get(0).get(0).toString()
+ assert executeSql.contains("slot_name") : "ExecuteSql should contain slot_name"
+ assert executeSql.contains("publication_name") : "ExecuteSql should contain publication_name"
+ assert executeSql.contains("doris_cdc_") : "slot_name should have doris_cdc_ prefix"
+ assert executeSql.contains("doris_pub_") : "publication_name should have doris_pub_ prefix"
+
+ // Wait for job to start running and create slot/publication on PG
+ try {
+ Awaitility.await().atMost(300, SECONDS)
+ .pollInterval(1, SECONDS).until(
+ {
+ def jobSuccendCount = sql """ select SucceedTaskCount from jobs("type"="insert") where Name = '${jobName}' and ExecuteType='STREAMING' """
+ log.info("jobSuccendCount: " + jobSuccendCount)
+ jobSuccendCount.size() == 1 && '2' <= jobSuccendCount.get(0).get(0)
Review Comment:
The Awaitility condition compares strings (`'2' <= jobSuccendCount...`). This is lexicographic and can behave incorrectly once the succeed count reaches multiple digits (e.g. "10"). Cast the query result to an integer/long and do a numeric comparison to avoid flaky waits.
```suggestion
if (jobSuccendCount.size() != 1) {
return false
}
def succeedTaskCount = jobSuccendCount.get(0).get(0).toString().toLong()
succeedTaskCount >= 2L
```
##########
regression-test/suites/job_p0/streaming_job/cdc/test_streaming_postgres_job_publication.groovy:
##########
@@ -0,0 +1,166 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+
+import org.awaitility.Awaitility
+
+import static java.util.concurrent.TimeUnit.SECONDS
+
+suite("test_streaming_postgres_job_publication", "p0,external,pg,external_docker,external_docker_pg,nondatalake") {
+ def jobName = "test_pg_pub_job"
+ def currentDb = (sql "select database()")[0][0]
+ def table1 = "pub_test_table1"
+ def table2 = "pub_test_table2"
+ def pgDB = "postgres"
+ def pgSchema = "cdc_test"
+ def pgUser = "postgres"
+ def pgPassword = "123456"
+
+ sql """DROP JOB IF EXISTS where jobname = '${jobName}'"""
+ sql """drop table if exists ${currentDb}.${table1} force"""
+ sql """drop table if exists ${currentDb}.${table2} force"""
+
+ String enabled = context.config.otherConfigs.get("enableJdbcTest")
+ if (enabled != null && enabled.equalsIgnoreCase("true")) {
+ String pg_port = context.config.otherConfigs.get("pg_14_port");
+ String externalEnvIp = context.config.otherConfigs.get("externalEnvIp")
+ String s3_endpoint = getS3Endpoint()
+ String bucket = getS3BucketName()
+ String driver_url = "https://${bucket}.${s3_endpoint}/regression/jdbc_driver/postgresql-42.5.0.jar"
+
+ // Setup PG tables
+ connect("${pgUser}", "${pgPassword}", "jdbc:postgresql://${externalEnvIp}:${pg_port}/${pgDB}") {
+ sql """DROP TABLE IF EXISTS ${pgDB}.${pgSchema}.${table1}"""
+ sql """DROP TABLE IF EXISTS ${pgDB}.${pgSchema}.${table2}"""
+ sql """CREATE TABLE ${pgDB}.${pgSchema}.${table1} (
+ "id" int PRIMARY KEY,
+ "name" varchar(200)
+ )"""
+ sql """CREATE TABLE ${pgDB}.${pgSchema}.${table2} (
+ "id" int PRIMARY KEY,
+ "name" varchar(200)
+ )"""
+ sql """INSERT INTO ${pgDB}.${pgSchema}.${table1} (id, name) VALUES (1, 'Alice');"""
+ sql """INSERT INTO ${pgDB}.${pgSchema}.${table2} (id, name) VALUES (1, 'Bob');"""
+ }
+
+ // ========== Test 1: auto-generated slot_name and publication_name ==========
+ sql """CREATE JOB ${jobName}
+ ON STREAMING
+ FROM POSTGRES (
+ "jdbc_url" = "jdbc:postgresql://${externalEnvIp}:${pg_port}/${pgDB}",
+ "driver_url" = "${driver_url}",
+ "driver_class" = "org.postgresql.Driver",
+ "user" = "${pgUser}",
+ "password" = "${pgPassword}",
+ "database" = "${pgDB}",
+ "schema" = "${pgSchema}",
+ "include_tables" = "${table1},${table2}",
+ "offset" = "initial"
+ )
+ TO DATABASE ${currentDb} (
+ "table.create.properties.replication_num" = "1"
+ )
+ """
+
+ // Verify SHOW contains auto-generated slot_name and publication_name
+ def jobInfo = sql """select ExecuteSql from jobs("type"="insert") where Name='${jobName}'"""
+ log.info("jobInfo ExecuteSql: " + jobInfo)
+ def executeSql = jobInfo.get(0).get(0).toString()
+ assert executeSql.contains("slot_name") : "ExecuteSql should contain slot_name"
+ assert executeSql.contains("publication_name") : "ExecuteSql should contain publication_name"
+ assert executeSql.contains("doris_cdc_") : "slot_name should have doris_cdc_ prefix"
+ assert executeSql.contains("doris_pub_") : "publication_name should have doris_pub_ prefix"
+
+ // Wait for job to start running and create slot/publication on PG
+ try {
+ Awaitility.await().atMost(300, SECONDS)
+ .pollInterval(1, SECONDS).until(
+ {
+ def jobSuccendCount = sql """ select SucceedTaskCount from jobs("type"="insert") where Name = '${jobName}' and ExecuteType='STREAMING' """
+ log.info("jobSuccendCount: " + jobSuccendCount)
+ jobSuccendCount.size() == 1 && '2' <= jobSuccendCount.get(0).get(0)
+ }
+ )
+ } catch (Exception ex){
+ def showjob = sql """select * from jobs("type"="insert") where Name='${jobName}'"""
+ def showtask = sql """select * from tasks("type"="insert") where JobName='${jobName}'"""
+ log.info("show job: " + showjob)
+ log.info("show task: " + showtask)
+ throw ex;
+ }
+
+ // Extract the actual publication name from ExecuteSql
+ def pubNameMatch = (executeSql =~ /publication_name'='([^']+)'/)
+ assert pubNameMatch.find() : "Should find publication_name in ExecuteSql"
+ def pubName = pubNameMatch.group(1)
+ log.info("Extracted publication name: " + pubName)
+
+ def slotNameMatch = (executeSql =~ /slot_name'='([^']+)'/)
+ assert slotNameMatch.find() : "Should find slot_name in ExecuteSql"
+ def slotName = slotNameMatch.group(1)
+ log.info("Extracted slot name: " + slotName)
+
+ // Verify on PG side: publication is NOT "FOR ALL TABLES"
+ connect("${pgUser}", "${pgPassword}", "jdbc:postgresql://${externalEnvIp}:${pg_port}/${pgDB}") {
+ def pubResult = sql """SELECT pubname, puballtables FROM pg_publication WHERE pubname = '${pubName}'"""
+ log.info("pg_publication: " + pubResult)
+ assert pubResult.size() == 1 : "Publication should exist"
+ assert pubResult[0][1] == false : "Publication should NOT be FOR ALL TABLES"
+
+ // Verify publication contains only the specified tables
+ def pubTables = sql """SELECT tablename FROM pg_publication_tables WHERE pubname = '${pubName}' ORDER BY tablename"""
+ log.info("pg_publication_tables: " + pubTables)
+ assert pubTables.size() == 2 : "Publication should contain exactly 2 tables"
+ def tableNames = pubTables.collect { it[0] }
+ assert tableNames.contains(table1) : "Publication should contain ${table1}"
+ assert tableNames.contains(table2) : "Publication should contain ${table2}"
+
+ // Verify replication slot exists
+ def slotResult = sql """SELECT slot_name, active FROM pg_replication_slots WHERE slot_name = '${slotName}'"""
+ log.info("pg_replication_slots: " + slotResult)
+ assert slotResult.size() == 1 : "Replication slot should exist"
+ }
+
+ // Check snapshot data synced correctly
+ qt_select_pub_snapshot_table1 """ SELECT * FROM ${table1} order by id asc """
+ qt_select_pub_snapshot_table2 """ SELECT * FROM ${table2} order by id asc """
+
+ // Drop job and verify PG resources are cleaned up
+ sql """DROP JOB IF EXISTS where jobname = '${jobName}'"""
+
+ sleep(5000) // wait for cleanup
+
+ connect("${pgUser}", "${pgPassword}", "jdbc:postgresql://${externalEnvIp}:${pg_port}/${pgDB}") {
+ def pubAfterDrop = sql """SELECT COUNT(1) FROM pg_publication WHERE pubname = '${pubName}'"""
+ log.info("pg_publication after drop: " + pubAfterDrop)
+ assert pubAfterDrop[0][0] == 0 : "Publication should be dropped after job deletion"
+
+ def slotAfterDrop = sql """SELECT COUNT(1) FROM pg_replication_slots WHERE slot_name = '${slotName}'"""
+ log.info("pg_replication_slots after drop: " + slotAfterDrop)
+ assert slotAfterDrop[0][0] == 0 : "Replication slot should be dropped after job deletion"
+ }
+
Review Comment:
The test uses a fixed `sleep(5000)` before asserting the publication/slot are gone. Cleanup can be asynchronous and this can be flaky on slower environments. Prefer polling with Awaitility until the PG queries return 0 (with a reasonable timeout) instead of a fixed sleep.
```suggestion
Awaitility.await()
.atMost(30, SECONDS)
.pollInterval(1, SECONDS)
.untilAsserted {
connect("${pgUser}", "${pgPassword}", "jdbc:postgresql://${externalEnvIp}:${pg_port}/${pgDB}") {
def pubAfterDrop = sql """SELECT COUNT(1) FROM pg_publication WHERE pubname = '${pubName}'"""
log.info("pg_publication after drop: " + pubAfterDrop)
assert pubAfterDrop[0][0] == 0 : "Publication should be dropped after job deletion"
def slotAfterDrop = sql """SELECT COUNT(1) FROM pg_replication_slots WHERE slot_name = '${slotName}'"""
log.info("pg_replication_slots after drop: " + slotAfterDrop)
assert slotAfterDrop[0][0] == 0 : "Replication slot should be dropped after job deletion"
}
}
```
##########
fs_brokers/cdc_client/src/main/java/org/apache/doris/cdcclient/source/reader/postgres/PostgresSourceReader.java:
##########
@@ -453,19 +461,50 @@ public Map<String, String> extractBinlogStateOffset(Object splitState) {
@Override
public void close(JobBaseConfig jobConfig) {
super.close(jobConfig);
- // drop pg slot
+ Map<String, String> config = jobConfig.getConfig();
try {
PostgresSourceConfig sourceConfig = getSourceConfig(jobConfig);
PostgresDialect dialect = new PostgresDialect(sourceConfig);
- String slotName = getSlotName(jobConfig.getJobId());
- LOG.info(
- "Dropping postgres replication slot {} for job {}",
- slotName,
- jobConfig.getJobId());
- dialect.removeSlot(slotName);
+
+ // Drop slot only if auto-created (doris_cdc_ prefix)
+ String slotName =
+ config.getOrDefault(
+ DataSourceConfigKeys.SLOT_NAME, getSlotName(jobConfig.getJobId()));
+ if (slotName.startsWith(DataSourceConfigKeys.DEFAULT_SLOT_PREFIX)) {
+ LOG.info(
+ "Dropping auto-created replication slot {} for job {}",
+ slotName,
+ jobConfig.getJobId());
+ dialect.removeSlot(slotName);
+ } else {
+ LOG.info(
+ "Skipping drop of user-provided replication slot {} for job {}",
+ slotName,
+ jobConfig.getJobId());
+ }
Review Comment:
Cleanup decides whether a slot/publication is “auto-created” solely by checking the prefix. This will incorrectly drop user-provided resources if the user chooses a name starting with the reserved prefix (contradicting the PR behavior guarantee). Either track an explicit “auto-created” flag in persisted job config, or reject user-provided names with these reserved prefixes during validation.
##########
fs_brokers/cdc_client/src/main/java/org/apache/doris/cdcclient/source/reader/postgres/PostgresSourceReader.java:
##########
@@ -453,19 +461,50 @@ public Map<String, String> extractBinlogStateOffset(Object splitState) {
@Override
public void close(JobBaseConfig jobConfig) {
super.close(jobConfig);
- // drop pg slot
+ Map<String, String> config = jobConfig.getConfig();
try {
PostgresSourceConfig sourceConfig = getSourceConfig(jobConfig);
PostgresDialect dialect = new PostgresDialect(sourceConfig);
- String slotName = getSlotName(jobConfig.getJobId());
- LOG.info(
- "Dropping postgres replication slot {} for job {}",
- slotName,
- jobConfig.getJobId());
- dialect.removeSlot(slotName);
+
+ // Drop slot only if auto-created (doris_cdc_ prefix)
+ String slotName =
+ config.getOrDefault(
+ DataSourceConfigKeys.SLOT_NAME, getSlotName(jobConfig.getJobId()));
+ if (slotName.startsWith(DataSourceConfigKeys.DEFAULT_SLOT_PREFIX)) {
+ LOG.info(
+ "Dropping auto-created replication slot {} for job {}",
+ slotName,
+ jobConfig.getJobId());
+ dialect.removeSlot(slotName);
+ } else {
+ LOG.info(
+ "Skipping drop of user-provided replication slot {} for job {}",
+ slotName,
+ jobConfig.getJobId());
+ }
+
+ // Drop publication only if auto-created (doris_pub_ prefix)
+ String pubName =
+ config.getOrDefault(
+ DataSourceConfigKeys.PUBLICATION_NAME,
+ "doris_pub_" + jobConfig.getJobId());
+ if (pubName.startsWith(DataSourceConfigKeys.DEFAULT_PUBLICATION_PREFIX)) {
+ LOG.info(
+ "Dropping auto-created publication {} for job {}",
+ pubName,
+ jobConfig.getJobId());
+ try (PostgresConnection connection = dialect.openJdbcConnection()) {
+ connection.execute("DROP PUBLICATION IF EXISTS " + pubName);
+ }
Review Comment:
`DROP PUBLICATION IF EXISTS ` is built via string concatenation without identifier quoting/escaping. If `publication_name` contains unexpected characters (or malicious content) this can fail or be exploited. Use a safe identifier-quoting utility (or a prepared/escaped execution path) and/or enforce strict identifier validation in FE (length/charset) before the value can reach here.
##########
fe/fe-core/src/main/java/org/apache/doris/job/extensions/insert/streaming/DataSourceConfigValidator.java:
##########
@@ -40,7 +40,9 @@ public class DataSourceConfigValidator {
DataSourceConfigKeys.SNAPSHOT_SPLIT_SIZE,
DataSourceConfigKeys.SNAPSHOT_PARALLELISM,
DataSourceConfigKeys.SSL_MODE,
- DataSourceConfigKeys.SSL_ROOTCERT
+ DataSourceConfigKeys.SSL_ROOTCERT,
+ DataSourceConfigKeys.SLOT_NAME,
+ DataSourceConfigKeys.PUBLICATION_NAME
);
Review Comment:
`slot_name` / `publication_name` are now accepted keys, but `validateSource()` does not validate them beyond non-empty. These values are later interpolated into PostgreSQL SQL (publication/slot creation & cleanup) without escaping/quoting, so invalid identifiers (e.g. hyphens/quotes) will break jobs and crafted values could enable SQL injection. Add explicit validation for these two keys (e.g. length <= 63, allowed characters, and reject quotes/whitespace); also consider rejecting values starting with the reserved auto-created prefixes if you rely on prefix-based cleanup.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improve](streaming-job) use per-table publication instead of ALL TABLES for PostgreSQL CDC [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62526:
URL: https://github.com/apache/doris/pull/62526#issuecomment-4257509604
# FE UT Coverage Report
Increment line coverage ` 0.00% (0/7)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62526_3fcc978b693bf9d873529667f3314d29372459c5/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62526_3fcc978b693bf9d873529667f3314d29372459c5/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improve](streaming-job) use per-table publication instead of ALL TABLES for PostgreSQL CDC [doris]
Posted by "JNSimba (via GitHub)" <gi...@apache.org>.
JNSimba commented on PR #62526:
URL: https://github.com/apache/doris/pull/62526#issuecomment-4265126654
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improve](streaming-job) use per-table publication instead of ALL TABLES for PostgreSQL CDC [doris]
Posted by "JNSimba (via GitHub)" <gi...@apache.org>.
JNSimba commented on PR #62526:
URL: https://github.com/apache/doris/pull/62526#issuecomment-4266028576
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improve](streaming-job) use per-table publication instead of ALL TABLES for PostgreSQL CDC [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #62526:
URL: https://github.com/apache/doris/pull/62526#discussion_r3091748792
##########
fs_brokers/cdc_client/src/main/java/org/apache/doris/cdcclient/source/reader/postgres/PostgresSourceReader.java:
##########
@@ -255,8 +287,116 @@ private PostgresSourceConfig generatePostgresConfig(
return configFactory.create(subtaskId);
}
- private String getSlotName(String jobId) {
- return "doris_cdc_" + jobId;
+ /**
+ * Resolve slot name from config; fall back to the legacy pre-PR default for jobs whose
+ * sourceProperties were persisted before the default-name populating logic was added.
+ */
+ private String resolveSlotName(Map<String, String> config, String jobId) {
+ return config.getOrDefault(
+ DataSourceConfigKeys.SLOT_NAME, DataSourceConfigKeys.DEFAULT_SLOT_PREFIX + jobId);
Review Comment:
Blocking: this is not the legacy publication name used by already persisted PostgreSQL CDC jobs. Before this PR the reader never set `publication.name`, so old jobs ran against the Debezium default `dbz_publication` with `ALL_TABLES`. After upgrade, jobs whose persisted `sourceProperties` do not contain `publication_name` will resolve to `doris_pub_<jobId>`, `isAutoGenerated()` becomes false, `publication.autocreate.mode` flips to `disabled`, and `initialize()` immediately validates a publication that does not exist. Existing jobs will fail to restart, and the legacy `doris_cdc_<jobId>` slot will also stop being cleaned up on drop. This needs an explicit compatibility branch or replay time backfill, not a new default here.
##########
fs_brokers/cdc_client/src/main/java/org/apache/doris/cdcclient/source/reader/postgres/PostgresSourceReader.java:
##########
@@ -453,21 +593,31 @@ public Map<String, String> extractBinlogStateOffset(Object splitState) {
@Override
public void close(JobBaseConfig jobConfig) {
super.close(jobConfig);
- // drop pg slot
Review Comment:
Blocking: this ownership check depends on the current slot and publication names, but those properties are still mutable through `ALTER JOB` because they are not rejected in `checkUnmodifiableSourceProperties`. A job created with Doris managed names can therefore be paused, have one of these names changed, and then hit this `return` on drop, leaving the original Doris created slot or publication behind. The same issue appears if Doris creates a newly supplied slot name and then later treats it as user owned. Ownership needs to be persisted when the resources are created, or these properties need to be immutable after creation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improve](streaming-job) use per-table publication instead of ALL TABLES for PostgreSQL CDC [doris]
Posted by "JNSimba (via GitHub)" <gi...@apache.org>.
JNSimba commented on PR #62526:
URL: https://github.com/apache/doris/pull/62526#issuecomment-4267585440
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improve](streaming-job) use per-table publication instead of ALL TABLES for PostgreSQL CDC [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62526:
URL: https://github.com/apache/doris/pull/62526#issuecomment-4260116916
# FE UT Coverage Report
Increment line coverage ` 20.83% (5/24)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62526_59a5e71c5430063548e8b53bdda684035979cbbe/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62526_59a5e71c5430063548e8b53bdda684035979cbbe/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improve](streaming-job) use per-table publication instead of ALL TABLES for PostgreSQL CDC [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62526:
URL: https://github.com/apache/doris/pull/62526#issuecomment-4278004593
# FE Regression Coverage Report
Increment line coverage ` 1.69% (2/118)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62526_8730818d26e7573135450b76ea7eb1b66dd3b0b0_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62526_8730818d26e7573135450b76ea7eb1b66dd3b0b0_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improve](streaming-job) use per-table publication instead of ALL TABLES for PostgreSQL CDC [doris]
Posted by "JNSimba (via GitHub)" <gi...@apache.org>.
JNSimba commented on PR #62526:
URL: https://github.com/apache/doris/pull/62526#issuecomment-4265129182
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improve](streaming-job) use per-table publication instead of ALL TABLES for PostgreSQL CDC [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62526:
URL: https://github.com/apache/doris/pull/62526#issuecomment-4260408394
# FE Regression Coverage Report
Increment line coverage ` 0.00% (0/25)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62526_59a5e71c5430063548e8b53bdda684035979cbbe_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62526_59a5e71c5430063548e8b53bdda684035979cbbe_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improve](streaming-job) use per-table publication instead of ALL TABLES for PostgreSQL CDC [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62526:
URL: https://github.com/apache/doris/pull/62526#issuecomment-4265899216
# FE Regression Coverage Report
Increment line coverage ` 0.00% (0/24)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62526_0fc7b73069673e831a1d8f572670afb9852a36f8_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62526_0fc7b73069673e831a1d8f572670afb9852a36f8_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improve](streaming-job) use per-table publication instead of ALL TABLES for PostgreSQL CDC [doris]
Posted by "JNSimba (via GitHub)" <gi...@apache.org>.
JNSimba commented on PR #62526:
URL: https://github.com/apache/doris/pull/62526#issuecomment-4257261032
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improve](streaming-job) use per-table publication instead of ALL TABLES for PostgreSQL CDC [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #62526:
URL: https://github.com/apache/doris/pull/62526#issuecomment-4268312615
OpenCode automated review failed and did not complete.
Error: Review step was failure (possibly timeout or cancelled)
Workflow run: https://github.com/apache/doris/actions/runs/24562607788
Please inspect the workflow logs and rerun the review after the underlying issue is resolved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improve](streaming-job) use per-table publication instead of ALL TABLES for PostgreSQL CDC [doris]
Posted by "JNSimba (via GitHub)" <gi...@apache.org>.
JNSimba commented on PR #62526:
URL: https://github.com/apache/doris/pull/62526#issuecomment-4277516772
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improve](streaming-job) use per-table publication instead of ALL TABLES for PostgreSQL CDC [doris]
Posted by "JNSimba (via GitHub)" <gi...@apache.org>.
JNSimba commented on PR #62526:
URL: https://github.com/apache/doris/pull/62526#issuecomment-4259588309
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improve](streaming-job) use per-table publication instead of ALL TABLES for PostgreSQL CDC [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62526:
URL: https://github.com/apache/doris/pull/62526#issuecomment-4266905608
# FE Regression Coverage Report
Increment line coverage ` 0.00% (0/101)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62526_f96507628f974e4bd48587f3e0bf169278cbdb24_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62526_f96507628f974e4bd48587f3e0bf169278cbdb24_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improve](streaming-job) use per-table publication instead of ALL TABLES for PostgreSQL CDC [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #62526:
URL: https://github.com/apache/doris/pull/62526#discussion_r3086250738
##########
fs_brokers/cdc_client/src/main/java/io/debezium/connector/postgresql/connection/PostgresReplicationConnection.java:
##########
@@ -248,14 +249,18 @@ private void createOrUpdatePublicationModeFilterted(
"No table filters found for filtered publication %s",
publicationName));
}
+ boolean supportPartitionRoot = !isUpdate
+ && ((BaseConnection) pgConnection()).haveMinimumServerVersion(ServerVersion.v13);
+ String pubViaRootSuffix = supportPartitionRoot
Review Comment:
`publication_name` now comes straight from user input, but this method still interpolates it directly into `CREATE/ALTER PUBLICATION` without any identifier quoting or syntax validation. That means values such as `MyPub` or `pub-name` are accepted by FE and then fail here at runtime, because PostgreSQL requires those identifiers to be quoted. Since `PostgresSourceReader.close()` also builds `DROP PUBLICATION IF EXISTS ` the same way, the issue affects both startup and cleanup. Please either validate `publication_name` against PostgreSQL's unquoted-identifier grammar before persisting it, or quote/escape it consistently at every SQL call site.
##########
fs_brokers/cdc_client/src/main/java/org/apache/doris/cdcclient/source/reader/postgres/PostgresSourceReader.java:
##########
@@ -453,19 +461,50 @@ public Map<String, String> extractBinlogStateOffset(Object splitState) {
@Override
public void close(JobBaseConfig jobConfig) {
super.close(jobConfig);
- // drop pg slot
+ Map<String, String> config = jobConfig.getConfig();
try {
PostgresSourceConfig sourceConfig = getSourceConfig(jobConfig);
PostgresDialect dialect = new PostgresDialect(sourceConfig);
- String slotName = getSlotName(jobConfig.getJobId());
- LOG.info(
- "Dropping postgres replication slot {} for job {}",
- slotName,
- jobConfig.getJobId());
- dialect.removeSlot(slotName);
Review Comment:
This prefix check does not actually tell us whether Doris created the resource. If a user explicitly sets `publication_name = doris_pub_shared` or `slot_name = doris_cdc_shared`, the PR description says those user-provided resources must be preserved, but `DROP JOB` will delete them here anyway. Please persist an explicit ownership flag (or equivalent metadata) when Doris auto-generates the slot/publication instead of inferring ownership from the name prefix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improve](streaming-job) use per-table publication instead of ALL TABLES for PostgreSQL CDC [doris]
Posted by "JNSimba (via GitHub)" <gi...@apache.org>.
JNSimba commented on PR #62526:
URL: https://github.com/apache/doris/pull/62526#issuecomment-4251742081
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improve](streaming-job) use per-table publication instead of ALL TABLES for PostgreSQL CDC [doris]
Posted by "JNSimba (via GitHub)" <gi...@apache.org>.
JNSimba merged PR #62526:
URL: https://github.com/apache/doris/pull/62526
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improve](streaming-job) use per-table publication instead of ALL TABLES for PostgreSQL CDC [doris]
Posted by "JNSimba (via GitHub)" <gi...@apache.org>.
JNSimba commented on PR #62526:
URL: https://github.com/apache/doris/pull/62526#issuecomment-4264953580
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improve](streaming-job) use per-table publication instead of ALL TABLES for PostgreSQL CDC [doris]
Posted by "JNSimba (via GitHub)" <gi...@apache.org>.
JNSimba commented on PR #62526:
URL: https://github.com/apache/doris/pull/62526#issuecomment-4267589483
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improve](streaming-job) use per-table publication instead of ALL TABLES for PostgreSQL CDC [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62526:
URL: https://github.com/apache/doris/pull/62526#issuecomment-4268431871
# FE Regression Coverage Report
Increment line coverage ` 1.71% (2/117)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62526_8c54460b9b6a624fa144b975e75eb850fa17265e_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62526_8c54460b9b6a624fa144b975e75eb850fa17265e_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improve](streaming-job) use per-table publication instead of ALL TABLES for PostgreSQL CDC [doris]
Posted by "JNSimba (via GitHub)" <gi...@apache.org>.
JNSimba commented on PR #62526:
URL: https://github.com/apache/doris/pull/62526#issuecomment-4266027370
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improve](streaming-job) use per-table publication instead of ALL TABLES for PostgreSQL CDC [doris]
Posted by "JNSimba (via GitHub)" <gi...@apache.org>.
JNSimba commented on PR #62526:
URL: https://github.com/apache/doris/pull/62526#issuecomment-4258358409
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improve](streaming-job) use per-table publication instead of ALL TABLES for PostgreSQL CDC [doris]
Posted by "JNSimba (via GitHub)" <gi...@apache.org>.
JNSimba commented on PR #62526:
URL: https://github.com/apache/doris/pull/62526#issuecomment-4278438417
run nonConcurrent
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improve](streaming-job) use per-table publication instead of ALL TABLES for PostgreSQL CDC [doris]
Posted by "Thearas (via GitHub)" <gi...@apache.org>.
Thearas commented on PR #62526:
URL: https://github.com/apache/doris/pull/62526#issuecomment-4251676975
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR).
Please clearly describe your PR:
1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org