You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "LeonYoah (via GitHub)" <gi...@apache.org> on 2026/01/31 07:48:44 UTC

[PR] Featrue/Add Kingbase Catalog Support [seatunnel]

LeonYoah opened a new pull request, #10427:
URL: https://github.com/apache/seatunnel/pull/10427

   <!--
   
   Thank you for contributing to SeaTunnel! Please make sure that your code changes
   are covered with tests. And in case of new features or big changes
   remember to adjust the documentation.
   
   Feel free to ping committers for the review!
   
   ## Contribution Checklist
     - Make sure that the pull request corresponds to a [GITHUB issue](https://github.com/apache/seatunnel/issues).
     - Name the pull request in the form "[Feature] [component] Title of the pull request", where *Feature* can be replaced by `Hotfix`, `Bug`, etc.
     - Minor fixes should be named following this pattern: `[hotfix] [docs] Fix typo in README.md doc`.
   -->
   
   ### Purpose of this pull request
   
   <!-- Describe the purpose of this pull request. For example: This pull request adds checkstyle plugin.-->
   Added support for the catalog functionality of the domestic database Kingbase, enabling metadata querying and auto table creation. Additionally, a related fix was implemented: When using query parameters in JDBC source, if the Kingbase instance is running in MySQL compatibility mode, the operation previously failed due to unsupported INT type errors. This occurred because the Kingbase JDBC driver's compatibility adaptations returned INT types, prompting this corrective update
   ### Does this PR introduce _any_ user-facing change?
   
   <!--
   Note that it means *any* user-facing change including all aspects such as the documentation fix.
   If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
   If possible, please also clarify if this is a user-facing change compared to the released SeaTunnel versions or within the unreleased branches such as dev.
   If no, write 'No'.
   If you are adding/modifying connector documents, please follow our new specifications: https://github.com/apache/seatunnel/issues/4544.
   -->
   
   
   ### How was this patch tested?
   
   <!--
   If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
   If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
   If tests were not added, please describe why they were not added and/or why it was difficult to add.
   If you are adding E2E test cases, maybe refer to https://github.com/apache/seatunnel/blob/dev/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-cdc-mysql-e2e/src/test/resources/mysqlcdc_to_mysql.conf, here is a good example.
   -->
   
   
   ### Check list
   
   * [ ] If any new Jar binary package adding in your PR, please add License Notice according
     [New License Guide](https://github.com/apache/seatunnel/blob/dev/docs/en/contribution/new-license.md)
   * [ ] If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs
   * [ ] If necessary, please update `incompatible-changes.md` to describe the incompatibility caused by this PR.
   * [ ] If you are contributing the connector code, please check that the following files are updated:
     1. Update [plugin-mapping.properties](https://github.com/apache/seatunnel/blob/dev/plugin-mapping.properties) and add new connector information in it
     2. Update the pom file of [seatunnel-dist](https://github.com/apache/seatunnel/blob/dev/seatunnel-dist/pom.xml)
     3. Add ci label in [label-scope-conf](https://github.com/apache/seatunnel/blob/dev/.github/workflows/labeler/label-scope-conf.yml)
     4. Add e2e testcase in [seatunnel-e2e](https://github.com/apache/seatunnel/tree/dev/seatunnel-e2e/seatunnel-connector-v2-e2e/)
     5. Update connector [plugin_config](https://github.com/apache/seatunnel/blob/dev/config/plugin_config)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature]Add Kingbase Catalog Support [seatunnel]

Posted by "zhangshenghang (via GitHub)" <gi...@apache.org>.
zhangshenghang commented on code in PR #10427:
URL: https://github.com/apache/seatunnel/pull/10427#discussion_r2757339439


##########
seatunnel-connectors-v2/connector-jdbc/src/test/resources/kingbase/license_39893_0.dat:
##########


Review Comment:
   Please obtain `KINGBASE_LICENSE` through environment variables



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature]Add Kingbase Catalog Support [seatunnel]

Posted by "LeonYoah (via GitHub)" <gi...@apache.org>.
LeonYoah commented on PR #10427:
URL: https://github.com/apache/seatunnel/pull/10427#issuecomment-4001634847

   > Don't Merge 
   
   Please review again.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature]Add Kingbase Catalog Support [seatunnel]

Posted by "LeonYoah (via GitHub)" <gi...@apache.org>.
LeonYoah commented on code in PR #10427:
URL: https://github.com/apache/seatunnel/pull/10427#discussion_r2757416978


##########
seatunnel-connectors-v2/connector-jdbc/src/test/resources/kingbase/license_39893_0.dat:
##########


Review Comment:
   Could you give an example? Which variable is it placed in? Does it need to be set in the GitHub repository, such as in GitHub Secrets?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature]Add Kingbase Catalog Support [seatunnel]

Posted by "LeonYoah (via GitHub)" <gi...@apache.org>.
LeonYoah commented on code in PR #10427:
URL: https://github.com/apache/seatunnel/pull/10427#discussion_r2780180076


##########
.github/workflows/backend.yml:
##########
@@ -399,6 +399,7 @@ jobs:
           ./mvnw -B -T 1 clean verify -DskipUT=false -DskipIT=true -D"license.skipAddThirdParty"=true -D"skip.ui"=true --no-snapshot-updates
         env:
           MAVEN_OPTS: -Xmx4096m
+          KINGBASE_LICENSE: ${{ vars.KINGBASE_LICENSE }}

Review Comment:
   ok



##########
.github/workflows/backend.yml:
##########
@@ -399,6 +399,7 @@ jobs:
           ./mvnw -B -T 1 clean verify -DskipUT=false -DskipIT=true -D"license.skipAddThirdParty"=true -D"skip.ui"=true --no-snapshot-updates
         env:
           MAVEN_OPTS: -Xmx4096m
+          KINGBASE_LICENSE: ${{ vars.KINGBASE_LICENSE }}

Review Comment:
   ok



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Featrue/Add Kingbase Catalog Support [seatunnel]

Posted by "davidzollo (via GitHub)" <gi...@apache.org>.
davidzollo commented on PR #10427:
URL: https://github.com/apache/seatunnel/pull/10427#issuecomment-3828547271

   
   Hi @LeonYoah, thanks for this contribution! Adding Catalog support for Kingbase is a great enhancement.
   
   After reviewing the code, I have a few suggestions to improve code consistency and compatibility:
   
   ## 1. Naming Inconsistency
   
   The existing classes in the project typically use `Kingbase` (only 'K' capitalized), such as `KingbaseDialect` and `KingbaseTypeMapper`.
   However, the new classes introduced in this PR use `KingBase` (CamelCase 'K' and 'B'), such as `KingBaseCatalog` and `KingBaseCatalogFactory`.
   
   **Suggestion:** Please rename the new classes to match the existing convention:
   - `KingBaseCatalog` -> `KingbaseCatalog`
   - `KingBaseCatalogFactory` -> `KingbaseCatalogFactory`
   - `KingBaseCreateTableSqlBuilder` -> `KingbaseCreateTableSqlBuilder`
   
   ## 2. Compatibility Risk with `rownum`
   
   In `KingBaseCatalog.java`:
   
   ```java
   @Override
   protected String getExistDataSql(TablePath tablePath) {
       return String.format(
               "select * from \"%s\".\"%s\" WHERE rownum = 1",
               tablePath.getSchemaName(), tablePath.getTableName());
   }
   ```
   
   **Risk:** `rownum` is an Oracle-specific syntax. While Kingbase supports an Oracle compatibility mode, it usually runs in Postgres compatibility mode by default, where `rownum` might not be supported or behaves differently. This could cause the catalog to fail in environments running in standard PG mode.
   
   **Suggestion:** Unless `rownum` is guaranteed to work in all Kingbase modes, it is safer to use the standard SQL standard or Postgres syntax: `LIMIT 1`.
   
   ```java
   "select * from \"%s\".\"%s\" LIMIT 1"
   ```
   
   ## 3. Documentation
   
   It would be helpful to update the `Features` section in `docs/en/connectors/source/Kingbase.md` to indicate that Kingbase now supports the Catalog feature.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature]Add Kingbase Catalog Support [seatunnel]

Posted by "LeonYoah (via GitHub)" <gi...@apache.org>.
LeonYoah commented on code in PR #10427:
URL: https://github.com/apache/seatunnel/pull/10427#discussion_r2780180142


##########
.github/workflows/backend.yml:
##########
@@ -399,6 +399,7 @@ jobs:
           ./mvnw -B -T 1 clean verify -DskipUT=false -DskipIT=true -D"license.skipAddThirdParty"=true -D"skip.ui"=true --no-snapshot-updates
         env:
           MAVEN_OPTS: -Xmx4096m
+          KINGBASE_LICENSE: ${{ vars.KINGBASE_LICENSE }}

Review Comment:
   ok



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature]Add Kingbase Catalog Support [seatunnel]

Posted by "chl-wxp (via GitHub)" <gi...@apache.org>.
chl-wxp commented on code in PR #10427:
URL: https://github.com/apache/seatunnel/pull/10427#discussion_r2752351774


##########
seatunnel-connectors-v2/connector-jdbc/src/test/java/org/apache/seatunnel/connectors/seatunnel/jdbc/catalog/kingbase/KingbaseCatalogTest.java:
##########
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.seatunnel.connectors.seatunnel.jdbc.catalog.kingbase;
+
+import org.apache.seatunnel.shade.com.google.common.collect.Lists;
+
+import org.apache.seatunnel.api.table.catalog.CatalogTable;
+import org.apache.seatunnel.api.table.catalog.TablePath;
+import org.apache.seatunnel.common.utils.JdbcUrlUtil;
+
+import org.junit.jupiter.api.AfterAll;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.Disabled;
+import org.junit.jupiter.api.Test;
+
+import java.util.List;
+
+@Disabled("Please Test it in your local environment")

Review Comment:
   Unit testing introduces containers, plus dialect and catalog testing, reference: https://github.com/apache/seatunnel/issues/10210



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature]Add Kingbase Catalog Support [seatunnel]

Posted by "yzeng1618 (via GitHub)" <gi...@apache.org>.
yzeng1618 commented on code in PR #10427:
URL: https://github.com/apache/seatunnel/pull/10427#discussion_r2752793495


##########
seatunnel-connectors-v2/connector-jdbc/src/main/java/org/apache/seatunnel/connectors/seatunnel/jdbc/catalog/kingbase/KingbaseCreateTableSqlBuilder.java:
##########
@@ -0,0 +1,143 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.seatunnel.connectors.seatunnel.jdbc.catalog.kingbase;
+
+import org.apache.seatunnel.shade.org.apache.commons.lang3.StringUtils;
+
+import org.apache.seatunnel.api.table.catalog.CatalogTable;
+import org.apache.seatunnel.api.table.catalog.Column;
+import org.apache.seatunnel.api.table.catalog.PrimaryKey;
+import org.apache.seatunnel.api.table.catalog.TablePath;
+import org.apache.seatunnel.connectors.seatunnel.jdbc.catalog.utils.CatalogUtils;
+import org.apache.seatunnel.connectors.seatunnel.jdbc.internal.dialect.DatabaseIdentifier;
+import org.apache.seatunnel.connectors.seatunnel.jdbc.internal.dialect.kingbase.KingbaseTypeConverter;
+
+import java.util.List;
+import java.util.UUID;
+import java.util.stream.Collectors;
+
+public class KingbaseCreateTableSqlBuilder {
+
+    private List<Column> columns;
+    private PrimaryKey primaryKey;
+    private String sourceCatalogName;
+    private String fieldIde;
+    private boolean createIndex;
+
+    public KingbaseCreateTableSqlBuilder(CatalogTable catalogTable, boolean createIndex) {
+        this.columns = catalogTable.getTableSchema().getColumns();
+        this.primaryKey = catalogTable.getTableSchema().getPrimaryKey();
+        this.sourceCatalogName = catalogTable.getCatalogName();
+        this.fieldIde = catalogTable.getOptions().get("fieldIde");
+        this.createIndex = createIndex;
+    }
+
+    public String build(TablePath tablePath) {
+        StringBuilder createTableSql = new StringBuilder();
+        createTableSql
+                .append("CREATE TABLE ")
+                .append(tablePath.getSchemaAndTableName("\""))
+                .append(" (\n");
+
+        List<String> columnSqls =
+                columns.stream()
+                        .map(column -> CatalogUtils.getFieldIde(buildColumnSql(column), fieldIde))
+                        .collect(Collectors.toList());
+
+        // Add primary key directly in the create table statement
+        if (createIndex
+                && primaryKey != null
+                && primaryKey.getColumnNames() != null
+                && primaryKey.getColumnNames().size() > 0) {
+            columnSqls.add(buildPrimaryKeySql(primaryKey));
+        }
+
+        createTableSql.append(String.join(",\n", columnSqls));
+        createTableSql.append("\n)");
+
+        List<String> commentSqls =
+                columns.stream()
+                        .filter(column -> StringUtils.isNotBlank(column.getComment()))
+                        .map(
+                                column ->
+                                        buildColumnCommentSql(
+                                                column, tablePath.getSchemaAndTableName("\"")))
+                        .collect(Collectors.toList());
+
+        if (!commentSqls.isEmpty()) {
+            createTableSql.append(";\n");
+            createTableSql.append(String.join(";\n", commentSqls));
+        }
+
+        return createTableSql.toString();
+    }
+
+    private String buildColumnSql(Column column) {
+        StringBuilder columnSql = new StringBuilder();
+        columnSql.append("\"").append(column.getName()).append("\" ");
+
+        String columnType =
+                StringUtils.equalsIgnoreCase(DatabaseIdentifier.KINGBASE, sourceCatalogName)
+                        ? column.getSourceType()
+                        : KingbaseTypeConverter.INSTANCE.reconvert(column).getColumnType();
+        columnSql.append(columnType);
+
+        if (!column.isNullable()) {
+            columnSql.append(" NOT NULL");
+        }
+
+        return columnSql.toString();
+    }
+
+    private String buildPrimaryKeySql(PrimaryKey primaryKey) {
+        String randomSuffix = UUID.randomUUID().toString().replace("-", "").substring(0, 4);
+        String columnNamesString =
+                primaryKey.getColumnNames().stream()
+                        .map(columnName -> "\"" + columnName + "\"")
+                        .collect(Collectors.joining(", "));
+
+        String primaryKeyStr = primaryKey.getPrimaryKey();
+        if (primaryKeyStr.length() > 25) {
+            primaryKeyStr = primaryKeyStr.substring(0, 25);
+        }
+
+        return CatalogUtils.getFieldIde(
+                "CONSTRAINT "
+                        + primaryKeyStr
+                        + "_"
+                        + randomSuffix
+                        + " PRIMARY KEY ("
+                        + columnNamesString
+                        + ")",
+                fieldIde);
+    }
+
+    private String buildColumnCommentSql(Column column, String tableName) {
+        StringBuilder columnCommentSql = new StringBuilder();
+        columnCommentSql
+                .append(CatalogUtils.quoteIdentifier("COMMENT ON COLUMN ", fieldIde))
+                .append(tableName)
+                .append(".");
+        columnCommentSql
+                .append(CatalogUtils.quoteIdentifier(column.getName(), fieldIde, "\""))
+                .append(CatalogUtils.quoteIdentifier(" IS '", fieldIde))
+                .append(column.getComment())
+                .append("'");

Review Comment:
   Unescaped single quotes in comment concatenation: it is recommended to use replace("'", "''").



##########
seatunnel-connectors-v2/connector-jdbc/src/main/java/org/apache/seatunnel/connectors/seatunnel/jdbc/catalog/kingbase/KingbaseCatalog.java:
##########
@@ -0,0 +1,261 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.seatunnel.connectors.seatunnel.jdbc.catalog.kingbase;
+
+import org.apache.seatunnel.shade.org.apache.commons.lang3.StringUtils;
+
+import org.apache.seatunnel.api.table.catalog.CatalogTable;
+import org.apache.seatunnel.api.table.catalog.Column;
+import org.apache.seatunnel.api.table.catalog.ConstraintKey;
+import org.apache.seatunnel.api.table.catalog.TablePath;
+import org.apache.seatunnel.api.table.catalog.exception.CatalogException;
+import org.apache.seatunnel.api.table.converter.BasicTypeDefine;
+import org.apache.seatunnel.common.exception.SeaTunnelRuntimeException;
+import org.apache.seatunnel.common.utils.JdbcUrlUtil;
+import org.apache.seatunnel.connectors.seatunnel.jdbc.catalog.AbstractJdbcCatalog;
+import org.apache.seatunnel.connectors.seatunnel.jdbc.catalog.utils.CatalogUtils;
+import org.apache.seatunnel.connectors.seatunnel.jdbc.internal.dialect.kingbase.KingbaseTypeConverter;
+import org.apache.seatunnel.connectors.seatunnel.jdbc.internal.dialect.kingbase.KingbaseTypeMapper;
+
+import lombok.extern.slf4j.Slf4j;
+
+import java.sql.Connection;
+import java.sql.DatabaseMetaData;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+
+import static org.apache.seatunnel.common.exception.CommonErrorCode.UNSUPPORTED_METHOD;
+
+@Slf4j
+public class KingbaseCatalog extends AbstractJdbcCatalog {
+
+    protected static List<String> EXCLUDED_SCHEMAS =
+            Collections.unmodifiableList(
+                    Arrays.asList(
+                            "INFORMATION_SCHEMA",
+                            "SYSAUDIT",
+                            "SYSLOGICAL",
+                            "SYS_CATALOG",
+                            "SYS_HM",
+                            "XLOG_RECORD_READ"));
+
+    private static final String SELECT_COLUMNS_SQL_TEMPLATE =
+            " SELECT \n"
+                    + "    a.attname AS column_name,\n"
+                    + "    CASE \n"
+                    + "        WHEN lower(format_type(a.atttypid, NULL)) IN ('varchar', 'character varying') THEN 'VARCHAR'\n"
+                    + "        WHEN lower(format_type(a.atttypid, NULL)) IN ('char', 'character') THEN 'CHAR'\n"
+                    + "        WHEN lower(format_type(a.atttypid, NULL)) IN ('boolean', 'bool') THEN 'BOOL'\n"
+                    + "        WHEN lower(format_type(a.atttypid, NULL)) = 'real' THEN 'FLOAT4'\n"
+                    + "        WHEN lower(format_type(a.atttypid, NULL)) = 'double precision' THEN 'FLOAT8'\n"
+                    + "        WHEN lower(format_type(a.atttypid, NULL)) = 'integer' THEN 'INT4'\n"
+                    + "        WHEN lower(format_type(a.atttypid, NULL)) = 'bigint' THEN 'INT8'\n"
+                    + "        WHEN lower(format_type(a.atttypid, NULL)) = 'smallint' THEN 'INT2'\n"
+                    + "        WHEN lower(format_type(a.atttypid, NULL)) = 'time without time zone' THEN 'TIME'\n"
+                    + "        WHEN lower(format_type(a.atttypid, NULL)) = 'timestamp without time zone' THEN 'TIMESTAMP'\n"
+                    + "        WHEN lower(format_type(a.atttypid, NULL)) = 'timestamp with time zone' THEN 'TIMESTAMPTZ'\n"
+                    + "        ELSE format_type(a.atttypid, NULL)\n"
+                    + "    END AS type_name,\n"
+                    + "    format_type(a.atttypid, a.atttypmod) AS full_type_name,\n"
+                    + "    CASE \n"
+                    + "        WHEN a.atttypid IN (SELECT oid FROM sys_type WHERE typname IN ( 'CHAR','CHARACTER','VARCHAR','CHARACTER VARYING','BPCHAR') )\n"
+                    + "        THEN ABS(a.atttypmod)     \n"
+                    + "        WHEN a.atttypid IN (SELECT oid FROM sys_type WHERE typname IN ('NUMERIC', 'DECIMAL'))\n"
+                    + "        THEN (a.atttypmod - 4) >> 16\n"
+                    + "        WHEN a.atttypid IN (SELECT oid FROM sys_type WHERE typname IN ('INT', 'INTEGER', 'SMALLINT', 'BIGINT'))\n"
+                    + "        THEN NULL\n"
+                    + "        WHEN a.atttypid IN (SELECT oid FROM sys_type WHERE typname IN ('TIME','TIMESTAMPTZ', 'TIMESTAMP'))\n"
+                    + "        THEN NULL\n"
+                    + "        ELSE NULL\n"
+                    + "    END AS column_length,\n"
+                    + "    CASE \n"
+                    + "        WHEN a.atttypid IN (SELECT oid FROM sys_type WHERE typname IN ('NUMERIC', 'DECIMAL'))\n"
+                    + "        THEN (a.atttypmod - 4) >> 16\n"
+                    + "        ELSE NULL\n"
+                    + "    END AS column_precision,\n"
+                    + "    CASE \n"
+                    + "        WHEN a.atttypid IN (SELECT oid FROM sys_type WHERE typname IN ('NUMERIC', 'DECIMAL'))\n"
+                    + "        THEN (a.atttypmod - 4) & 65535\n"
+                    + "        ELSE NULL\n"
+                    + "    END AS column_scale,\n"
+                    + "    d.description AS column_comment,\n"
+                    + "    pg_get_expr(ad.adbin, ad.adrelid) AS default_value,\n"
+                    + "    CASE \n"
+                    + "        WHEN a.attnotnull = false THEN 'YES'\n"
+                    + "        ELSE 'NO'\n"
+                    + "    END AS is_nullable\n"
+                    + "FROM \n"
+                    + "    sys_class c\n"
+                    + "    JOIN sys_namespace n ON c.relnamespace = n.oid\n"
+                    + "    JOIN sys_attribute a ON a.attrelid = c.oid\n"
+                    + "    LEFT JOIN sys_description d ON d.objoid = a.attrelid AND d.objsubid = a.attnum\n"
+                    + "    LEFT JOIN sys_attrdef ad ON ad.adrelid = a.attrelid AND ad.adnum = a.attnum\n"
+                    + "WHERE \n"
+                    + "    n.nspname = '%s' \n"
+                    + "    AND c.relname = '%s' \n"
+                    + "    AND a.attnum > 0 \n"
+                    + "    AND NOT a.attisdropped;";
+
+    public KingbaseCatalog(
+            String catalogName,
+            String username,
+            String pwd,
+            JdbcUrlUtil.UrlInfo urlInfo,
+            String defaultSchema,
+            String driverClass) {
+        super(catalogName, username, pwd, urlInfo, defaultSchema, driverClass);
+    }
+
+    @Override
+    protected String getListDatabaseSql() {
+        return "SELECT current_database();";
+    }
+
+    /**
+     * Override the databaseExists method because SELECT current_database() does not support WHERE
+     */
+    @Override
+    public boolean databaseExists(String databaseName) throws CatalogException {
+        if (StringUtils.isBlank(databaseName)) {
+            return false;
+        }
+        try {
+            return querySQLResultExists(getUrlFromDatabaseName(databaseName), getListDatabaseSql());
+        } catch (SeaTunnelRuntimeException e) {
+            if (e.getSeaTunnelErrorCode().getCode().equals(UNSUPPORTED_METHOD.getCode())) {
+                log.warn(
+                        "The catalog: {} is not supported the getListDatabaseSql for databaseExists",
+                        this.catalogName);
+                return listDatabases().contains(databaseName);
+            }
+            throw e;
+        } catch (SQLException e) {
+            throw new CatalogException("查询数据库是否存在失败: " + databaseName, e);
+        }
+    }
+
+    @Override
+    protected String getCreateTableSql(
+            TablePath tablePath, CatalogTable table, boolean createIndex) {
+        return new KingbaseCreateTableSqlBuilder(table, createIndex).build(tablePath);
+    }
+
+    @Override
+    protected String getDropTableSql(TablePath tablePath) {
+        return String.format("DROP TABLE %s", tablePath.getSchemaAndTableName("\""));
+    }
+
+    @Override
+    protected String getListTableSql(String databaseName) {
+        return "SELECT SCHEMANAME ,TABLENAME FROM SYS_TABLES";
+    }
+
+    @Override
+    protected String getTableWithConditionSql(TablePath tablePath) {
+        return String.format(
+                getListTableSql(tablePath.getDatabaseName())
+                        + "  where SCHEMANAME = '%s' and TABLENAME = '%s';",
+                tablePath.getSchemaName(),
+                tablePath.getTableName());
+    }
+
+    @Override
+    protected String getTableName(ResultSet rs) throws SQLException {
+        if (EXCLUDED_SCHEMAS.contains(rs.getString(1))) {
+            return null;
+        }
+        return rs.getString(1) + "." + rs.getString(2);
+    }
+
+    @Override
+    protected String getSelectColumnsSql(TablePath tablePath) {
+        return String.format(
+                SELECT_COLUMNS_SQL_TEMPLATE, tablePath.getSchemaName(), tablePath.getTableName());
+    }
+
+    @Override
+    protected Column buildColumn(ResultSet resultSet) throws SQLException {
+        String columnName = resultSet.getString("COLUMN_NAME");
+        String typeName = resultSet.getString("TYPE_NAME");
+        String fullTypeName = resultSet.getString("FULL_TYPE_NAME");
+        long columnLength = resultSet.getLong("COLUMN_LENGTH");
+        long columnPrecision = resultSet.getLong("COLUMN_PRECISION");
+        int columnScale = resultSet.getInt("COLUMN_SCALE");
+        String columnComment = resultSet.getString("COLUMN_COMMENT");
+        Object defaultValue = resultSet.getObject("DEFAULT_VALUE");
+        boolean isNullable = resultSet.getString("IS_NULLABLE").equals("YES");
+
+        BasicTypeDefine typeDefine =
+                BasicTypeDefine.builder()
+                        .name(columnName)
+                        .columnType(typeName)

Review Comment:
   FULL_TYPE_NAME is read but not used; columnType/sourceType loses information such as VARCHAR(255), CHAR(10), and NUMERIC(38,18).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Featrue/Add Kingbase Catalog Support [seatunnel]

Posted by "DanielCarter-stack (via GitHub)" <gi...@apache.org>.
DanielCarter-stack commented on PR #10427:
URL: https://github.com/apache/seatunnel/pull/10427#issuecomment-3831169266

   <!-- code-pr-reviewer -->
   <!-- cpr:pr_reply_v2_parts {"group": "apache/seatunnel#10427", "part": 1, "total": 1} -->
   ### Issue 1: All Tests Disabled
   
   **Location**: `seatunnel-connectors-v2/connector-jdbc/src/test/java/org/apache/seatunnel/connectors/seatunnel/jdbc/catalog/kingbase/KingbaseCatalogTest.java:21`
   
   ```java
   @Disabled("Please Test it in your local environment")
   class KingbaseCatalogTest {
       // All test methods are disabled
   }
   ```
   
   **Problem Description**:
   The test class is disabled by the `@Disabled` annotation, preventing CI from running these tests. This violates the "covered with tests" promise in the PR Checklist.
   
   **Potential Risks**:
   - Unable to verify code correctness in CI
   - Potential regression issues cannot be detected in a timely manner
   - Reduced code quality assurance
   
   **Scope of Impact**:
   - Direct impact: KingbaseCatalog, KingbaseCreateTableSqlBuilder
   - Indirect impact: All features depending on Kingbase Catalog
   - Affected area: Kingbase Connector
   
   **Severity**: CRITICAL
   
   **Improvement Suggestions**:
   ```java
   // Remove @Disabled or use Testcontainers
   @Testcontainers
   class KingbaseCatalogTest {
       @Container
       private static final PostgreSQLContainer<?> kingbaseContainer =
           new PostgreSQLContainer<>("kingbase/kes:latest")
               .withDatabaseName("test")
               .withUsername("kingbase")
               .withPassword("kingbase");
       
       @Test
       void databaseExists() {
           // Test using real containers
       }
   }
   ```
   
   **Reason**: Apache projects require all code changes to be covered by executable tests.
   
   ---
   
   ### Issue 2: Hardcoded Chinese Exception Messages
   
   **Location**: `seatunnel-connectors-v2/connector-jdbc/src/main/java/org/apache/seatunnel/connectors/seatunnel/jdbc/catalog/kingbase/KingbaseCatalog.java:146`
   
   ```java
   } catch (SQLException e) {
       throw new CatalogException("查询数据库是否存在失败: " + databaseName, e);
   }
   ```
   
   **Problem Description**:
   Using hardcoded Chinese strings as exception messages does not comply with internationalization standards. SeaTunnel is an international project, and error messages should be in English.
   
   **Related Context**:
   - Parent class `AbstractJdbcCatalog` uses English error messages
   - Other Catalog implementations (such as `PostgresCatalog`) use English
   
   **Potential Risks**:
   - International users cannot understand error messages
   - Difficult log analysis
   - Inconsistent with project standards
   
   **Scope of Impact**:
   - Direct impact: KingbaseCatalog.databaseExists()
   - Indirect impact: All user code calling this method
   - Affected area: Single Connector
   
   **Severity**: MAJOR
   
   **Improvement Suggestions**:
   ```java
   } catch (SQLException e) {
       throw new CatalogException(
           String.format("Failed to check if database exists: %s", databaseName), 
           e
       );
   }
   ```
   
   **Reason**: Complies with Apache project internationalization standards.
   
   ---
   
   ### Issue 3: Missing Null Check
   
   **Location**: `seatunnel-connectors-v2/connector-jdbc/src/main/java/org/apache/seatunnel/connectors/seatunnel/jdbc/catalog/kingbase/KingbaseCatalog.java:196-214`
   
   ```java
   @Override
   protected Column buildColumn(ResultSet resultSet) throws SQLException {
       String columnName = resultSet.getString("COLUMN_NAME");
       String typeName = resultSet.getString("TYPE_NAME");
       String fullTypeName = resultSet.getString("FULL_TYPE_NAME");
       // ...
       Object defaultValue = resultSet.getObject("DEFAULT_VALUE");
       boolean isNullable = resultSet.getString("IS_NULLABLE").equals("YES");
       // ...
   }
   ```
   
   **Problem Description**:
   There is no null check on the return value of `getString()`. If certain columns in SQL query results are NULL, it will cause `NullPointerException`.
   
   **Related Context**:
   - `SELECT_COLUMNS_SQL_TEMPLATE` uses `LEFT JOIN`, which may produce NULL values
   - Especially `column_comment` and `default_value` fields may be NULL
   
   **Potential Risks**:
   - When querying columns containing NULL values, NPE will be thrown
   - Users cannot retrieve table metadata
   - Job failure
   
   **Scope of Impact**:
   - Direct impact: KingbaseCatalog.getTable()
   - Indirect impact: All operations using Catalog API to query table metadata
   - Affected area: Single Connector
   
   **Severity**: MAJOR
   
   **Improvement Suggestions**:
   ```java
   @Override
   protected Column buildColumn(ResultSet resultSet) throws SQLException {
       String columnName = resultSet.getString("COLUMN_NAME");
       if (columnName == null) {
           throw new SQLException("COLUMN_NAME cannot be null");
       }
       
       String typeName = resultSet.getString("TYPE_NAME");
       String fullTypeName = resultSet.getString("FULL_TYPE_NAME");
       long columnLength = resultSet.getLong("COLUMN_LENGTH");
       long columnPrecision = resultSet.getLong("COLUMN_PRECISION");
       int columnScale = resultSet.getInt("COLUMN_SCALE");
       
       String columnComment = resultSet.getString("COLUMN_COMMENT");
       Object defaultValue = resultSet.getObject("DEFAULT_VALUE");
       
       String isNullableStr = resultSet.getString("IS_NULLABLE");
       boolean isNullable = "YES".equalsIgnoreCase(isNullableStr);
       
       BasicTypeDefine typeDefine = BasicTypeDefine.builder()
               .name(columnName)
               .columnType(typeName)
               .dataType(typeName)
               .length(columnLength)
               .precision(columnPrecision)
               .scale(columnScale)
               .nullable(isNullable)
               .defaultValue(defaultValue)
               .comment(columnComment)
               .build();
       return KingbaseTypeConverter.INSTANCE.convert(typeDefine);
   }
   ```
   
   **Reason**: Defensive programming to avoid NPE.
   
   ---
   
   ### Issue 4: SQL Injection Risk (Potential)
   
   **Location**: `seatunnel-connectors-v2/connector-jdbc/src/main/java/org/apache/seatunnel/connectors/seatunnel/jdbc/catalog/kingbase/KingbaseCatalog.java:189-192`
   
   ```java
   @Override
   protected String getSelectColumnsSql(TablePath tablePath) {
       return String.format(
               SELECT_COLUMNS_SQL_TEMPLATE, 
               tablePath.getSchemaName(), 
               tablePath.getTableName());
   }
   ```
   
   **Problem Description**:
   Using `String.format` to directly concatenate user input into SQL poses SQL injection risks. Although there are single quotes wrapping it in `SELECT_COLUMNS_SQL_TEMPLATE`, the input is not escaped.
   
   **Related Context**:
   - Other implementations of `AbstractJdbcCatalog` (such as `PostgresCatalog`) also use the same pattern
   - This is a systemic issue, not specific to this PR
   - However, this PR introduces this risk point
   
   **Potential Risks**:
   - If schema or table names contain single quotes, SQL syntax errors will occur
   - Malicious users may construct special table names for SQL injection attacks
   
   **Scope of Impact**:
   - Direct impact: KingbaseCatalog.getSelectColumnsSql()
   - Indirect impact: buildColumn(), getTable()
   - Affected area: Single Connector
   
   **Severity**: MAJOR
   
   **Improvement Suggestions**:
   ```java
   @Override
   protected String getSelectColumnsSql(TablePath tablePath) {
       String schemaName = escapeSqlIdentifier(tablePath.getSchemaName());
       String tableName = escapeSqlIdentifier(tablePath.getTableName());
       return String.format(SELECT_COLUMNS_SQL_TEMPLATE, schemaName, tableName);
   }
   
   private String escapeSqlIdentifier(String identifier) {
       return identifier.replace("'", "''");
   }
   ```
   
   **Reason**: Prevent SQL injection, comply with secure coding standards.
   
   ---
   
   ### Issue 5: KB_CLOB Type Sets columnLength Twice
   
   **Location**: `seatunnel-connectors-v2/connector-jdbc/src/main/java/org/apache/seatunnel/connectors/seatunnel/jdbc/internal/dialect/kingbase/KingbaseTypeConverter.java:190-194`
   
   ```java
   case KB_CLOB:
       builder.dataType(BasicType.STRING_TYPE);
       builder.columnLength(typeDefine.getLength());
       builder.columnLength((long) (1024 * 1024 * 1024));  // Second setup
       break;
   ```
   
   **Problem Description**:
   `columnLength` is set twice, the second time will overwrite the first value, causing `typeDefine.getLength()` to be ignored.
   
   **Related Context**:
   - `builder` is Lombok's `@Builder` pattern
   - The second call will overwrite the first value
   - This is an obvious copy-paste error
   
   **Potential Risks**:
   - CLOB type length is always set to 1GB
   - Ignores the actual length defined in the database
   - May lead to inaccurate type mapping
   
   **Scope of Impact**:
   - Direct impact: KB_CLOB type conversion
   - Indirect impact: CLOB column metadata
   - Affected area: Single Connector
   
   **Severity**: MINOR
   
   **Improvement Suggestions**:
   ```java
   case KB_CLOB:
       builder.dataType(BasicType.STRING_TYPE);
       if (typeDefine.getLength() != null && typeDefine.getLength() > 0) {
           builder.columnLength(typeDefine.getLength());
       } else {
           builder.columnLength((long) (1024 * 1024 * 1024));
       }
       break;
   ```
   
   **Reason**: Fix logic error, preserve user-defined length.
   
   ---
   
   ### Issue 6: MySqlTypeConverter Constant Visibility Modification Affects Encapsulation
   
   **Location**: `seatunnel-connectors-v2/connector-jdbc/src/main/java/org/apache/seatunnel/connectors/seatunnel/jdbc/internal/dialect/mysql/MySqlTypeConverter.java:45-98`
   
   ```java
   // Before modification
   -static final String MYSQL_INT = "INT";
   -static final String MYSQL_DATETIME = "DATETIME";
   // ... 38 constants
   
   // After modification
   +public static final String MYSQL_INT = "INT";
   +public static final String MYSQL_DATETIME = "DATETIME";
   // ... 38 constants
   ```
   
   **Problem Description**:
   Changing 38 private constants to public constants increases API surface area and breaks encapsulation. Although this solves Kingbase's needs, it is not the optimal solution.
   
   **Related Context**:
   - Referenced location: KingbaseTypeConverter.java:73-116
   - Caller: `case MySqlTypeConverter.MYSQL_INT:`
   - Similar references: OracleTypeConverter.ORACLE_NUMBER, SqlServerTypeConverter.SQLSERVER_DATETIME2
   
   **Potential Risks**:
   - These constants now become part of the public API
   - Modifying these constants will affect all callers
   - Increases maintenance burden for MySQL, Oracle, and SQL Server implementations
   - Violates the "Principle of Least Knowledge"
   
   **Scope of Impact**:
   - Direct impact: MySqlTypeConverter, OracleTypeConverter, SqlServerTypeConverter
   - Indirect impact: Other code that may depend on these constants in the future
   - Affected area: Multiple Connectors
   
   **Severity**: MINOR
   
   **Improvement Suggestions**:
   ```java
   // Create new file: JdbcCompatibilityTypeConstants.java
   package org.apache.seatunnel.connectors.seatunnel.jdbc.internal.dialect;
   
   public final class JdbcCompatibilityTypeConstants {
       // MySQL compatibility types
       public static final String MYSQL_INT = "INT";
       public static final String MYSQL_DATETIME = "DATETIME";
       // ...
       
       // Oracle compatibility types
       public static final String ORACLE_NUMBER = "NUMBER";
       // ...
       
       // SQLServer compatibility types
       public static final String SQLSERVER_DATETIME2 = "DATETIME2";
       // ...
       
       private JdbcCompatibilityTypeConstants() {}
   }
   ```
   
   Then reference in Kingbase:
   ```java
   import static org.apache.seatunnel.connectors.seatunnel.jdbc.internal.dialect.JdbcCompatibilityTypeConstants.*;
   
   case MYSQL_INT:
       builder.dataType(BasicType.INT_TYPE);
       break;
   ```
   
   **Reason**: Centrally manage compatibility type constants, reduce coupling, improve maintainability.
   
   ---
   
   ### Issue 8: seatunnel-dist/pom.xml Not Updated
   
   **Location**: `seatunnel-dist/pom.xml`
   
   **Problem Description**:
   The PR did not update the `seatunnel-dist/pom.xml` file. According to the Checklist, when adding a new connector, this file needs to be updated.
   
   **Related Context**:
   - Checklist requirement: "Update the pom file of seatunnel-dist"
   - Kingbase uses the existing connector-jdbc, so may not need independent dependencies
   
   **Potential Risks**:
   - If Kingbase requires specific JDBC driver dependencies, they need to be declared in the dist pom
   - May result in releases missing necessary drivers
   
   **Scope of Impact**:
   - Direct impact: SeaTunnel release
   - Indirect impact: Users cannot use Kingbase after downloading the release
   - Affected area: Build and distribution
   
   **Severity**: MAJOR
   
   **Improvement Suggestions**:
   Check if Kingbase JDBC driver dependency needs to be added:
   ```xml
   <!-- 在 seatunnel-dist/pom.xml 中 -->
   <dependency>
       <groupId>com.kingbase</groupId>
       <artifactId>kingbase8</artifactId>
       <version>${kingbase.driver.version}</version>
   </dependency>
   ```
   
   **Reason**: Ensure releases include all necessary dependencies.
   
   ---
   
   ### Issue 9: Incomplete Documentation
   
   **Location**: `docs/en/connectors/source/Kingbase.md`, `docs/zh/connectors/source/Kingbase.md`
   
   **Problem Description**:
   The documentation only adds Connector configuration items, without explaining how to use Catalog functionality, supported data type mappings, and special handling for MySQL compatibility mode.
   
   **Related Context**:
   - PR title: "Add Kingbase Catalog Support"
   - Documentation only updates the Source configuration section
   - Missing Catalog API usage examples
   
   **Potential Risks**:
   - Users don't know how to use Kingbase Catalog functionality
   - Missing type mapping table may cause usage confusion
   - Special handling for MySQL compatibility mode is not documented
   
   **Scope of Impact**:
   - Direct impact: Kingbase users
   - Indirect impact: User support and troubleshooting
   - Affected area: Kingbase Connector
   
   **Severity**: MAJOR
   
   **Improvement Suggestions**:
   Add in `docs/en/connectors/source/Kingbase.md`:
   ```markdown
   ## Catalog Support
   
   Kingbase connector supports catalog functionality for metadata querying and automatic table creation.
   
   ### Supported Data Types
   
   | Kingbase Type | SeaTunnel Type | Notes |
   |---------------|----------------|-------|
   | INT | INT | MySQL compatibility mode |
   | INTEGER | INT | Native type |
   | VARCHAR | STRING | |
   | NUMBER | DECIMAL | Oracle compatibility mode |
   | ... | ... | |
   
   ### MySQL Compatibility Mode
   
   When Kingbase is running in MySQL compatibility mode, the connector will automatically map MySQL types (e.g., INT, DATETIME) to appropriate SeaTunnel types.
   
   ### Example
   ```hocon
   source {
     KingbaseCatalog {
       catalog {
         database = "test_db"
         schema = "public"
       }
       table_path = "public.my_table"
     }
   }
   ```
   ```
   
   ** Reason**: Provide complete usage documentation to reduce user learning costs.
   
   ---
   
   # ## Issue 10: label-scope-conf.yml not updated
   
   ** Location**: `.github/workflows/labeler/label-scope-conf.yml`
   
   ** Problem Description**:
   PR 没有更新 `label-scope-conf.yml` 文件。根据 Checklist,添加新 connector 时需要添加 CI label。
   
   ** Related Context**:
   - Checklist 要求: "Add ci label in label-scope-conf.yml"
   - Kingbase 是新添加的 Catalog 支持
   - 需要 CI 系统能够识别 Kingbase 相关的 PR
   
   ** Potential Risks**:
   - Kingbase 相关的 PR 无法自动打上 label
   - 可能导致 CI 流程不正确
   
   ** Impact Scope**:
   - 直接影响: GitHub Actions CI
   - 间接影响: PR 自动化处理
   - 影响面: CI/CD 流程
   
   ** Severity**: MINOR
   
   ** Improvement Suggestions**:
   在 `.github/workflows/labeler/label-scope-conf.yml` 中添加:
   ```yaml
   connector-jdbc:
     - connector-jdbc/**/*
     - "**/*kingbase*"
   ```
   
   **Reason**: Complies with PR Checklist requirements, ensures correct CI operation.
   
   ---


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature]Add Kingbase Catalog Support [seatunnel]

Posted by "zhangshenghang (via GitHub)" <gi...@apache.org>.
zhangshenghang commented on code in PR #10427:
URL: https://github.com/apache/seatunnel/pull/10427#discussion_r2779302119


##########
.github/workflows/backend.yml:
##########
@@ -399,6 +399,7 @@ jobs:
           ./mvnw -B -T 1 clean verify -DskipUT=false -DskipIT=true -D"license.skipAddThirdParty"=true -D"skip.ui"=true --no-snapshot-updates
         env:
           MAVEN_OPTS: -Xmx4096m
+          KINGBASE_LICENSE: ${{ vars.KINGBASE_LICENSE }}

Review Comment:
   If it is not used, please delete it



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature]Add Kingbase Catalog Support [seatunnel]

Posted by "davidzollo (via GitHub)" <gi...@apache.org>.
davidzollo merged PR #10427:
URL: https://github.com/apache/seatunnel/pull/10427


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature]Add Kingbase Catalog Support [seatunnel]

Posted by "LeonYoah (via GitHub)" <gi...@apache.org>.
LeonYoah commented on code in PR #10427:
URL: https://github.com/apache/seatunnel/pull/10427#discussion_r2887174323


##########
seatunnel-connectors-v2/connector-jdbc/src/test/java/org/apache/seatunnel/connectors/seatunnel/jdbc/catalog/kingbase/KingbaseCatalogTest.java:
##########
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.seatunnel.connectors.seatunnel.jdbc.catalog.kingbase;
+
+import org.apache.seatunnel.shade.com.google.common.collect.Lists;
+
+import org.apache.seatunnel.api.table.catalog.CatalogTable;
+import org.apache.seatunnel.api.table.catalog.TablePath;
+import org.apache.seatunnel.common.utils.JdbcUrlUtil;
+
+import org.junit.jupiter.api.AfterAll;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.Disabled;
+import org.junit.jupiter.api.Test;
+
+import java.util.List;
+
+@Disabled("Please Test it in your local environment")

Review Comment:
   > Unit testing introduces containers, plus dialect and catalog testing, reference: https://github.com/apache/seatunnel/issues/10210
   
   Please review again.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature]Add Kingbase Catalog Support [seatunnel]

Posted by "LeonYoah (via GitHub)" <gi...@apache.org>.
LeonYoah commented on code in PR #10427:
URL: https://github.com/apache/seatunnel/pull/10427#discussion_r2752819106


##########
seatunnel-connectors-v2/connector-jdbc/src/main/java/org/apache/seatunnel/connectors/seatunnel/jdbc/catalog/kingbase/KingbaseCreateTableSqlBuilder.java:
##########
@@ -0,0 +1,143 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.seatunnel.connectors.seatunnel.jdbc.catalog.kingbase;
+
+import org.apache.seatunnel.shade.org.apache.commons.lang3.StringUtils;
+
+import org.apache.seatunnel.api.table.catalog.CatalogTable;
+import org.apache.seatunnel.api.table.catalog.Column;
+import org.apache.seatunnel.api.table.catalog.PrimaryKey;
+import org.apache.seatunnel.api.table.catalog.TablePath;
+import org.apache.seatunnel.connectors.seatunnel.jdbc.catalog.utils.CatalogUtils;
+import org.apache.seatunnel.connectors.seatunnel.jdbc.internal.dialect.DatabaseIdentifier;
+import org.apache.seatunnel.connectors.seatunnel.jdbc.internal.dialect.kingbase.KingbaseTypeConverter;
+
+import java.util.List;
+import java.util.UUID;
+import java.util.stream.Collectors;
+
+public class KingbaseCreateTableSqlBuilder {
+
+    private List<Column> columns;
+    private PrimaryKey primaryKey;
+    private String sourceCatalogName;
+    private String fieldIde;
+    private boolean createIndex;
+
+    public KingbaseCreateTableSqlBuilder(CatalogTable catalogTable, boolean createIndex) {
+        this.columns = catalogTable.getTableSchema().getColumns();
+        this.primaryKey = catalogTable.getTableSchema().getPrimaryKey();
+        this.sourceCatalogName = catalogTable.getCatalogName();
+        this.fieldIde = catalogTable.getOptions().get("fieldIde");
+        this.createIndex = createIndex;
+    }
+
+    public String build(TablePath tablePath) {
+        StringBuilder createTableSql = new StringBuilder();
+        createTableSql
+                .append("CREATE TABLE ")
+                .append(tablePath.getSchemaAndTableName("\""))
+                .append(" (\n");
+
+        List<String> columnSqls =
+                columns.stream()
+                        .map(column -> CatalogUtils.getFieldIde(buildColumnSql(column), fieldIde))
+                        .collect(Collectors.toList());
+
+        // Add primary key directly in the create table statement
+        if (createIndex
+                && primaryKey != null
+                && primaryKey.getColumnNames() != null
+                && primaryKey.getColumnNames().size() > 0) {
+            columnSqls.add(buildPrimaryKeySql(primaryKey));
+        }
+
+        createTableSql.append(String.join(",\n", columnSqls));
+        createTableSql.append("\n)");
+
+        List<String> commentSqls =
+                columns.stream()
+                        .filter(column -> StringUtils.isNotBlank(column.getComment()))
+                        .map(
+                                column ->
+                                        buildColumnCommentSql(
+                                                column, tablePath.getSchemaAndTableName("\"")))
+                        .collect(Collectors.toList());
+
+        if (!commentSqls.isEmpty()) {
+            createTableSql.append(";\n");
+            createTableSql.append(String.join(";\n", commentSqls));
+        }
+
+        return createTableSql.toString();
+    }
+
+    private String buildColumnSql(Column column) {
+        StringBuilder columnSql = new StringBuilder();
+        columnSql.append("\"").append(column.getName()).append("\" ");
+
+        String columnType =
+                StringUtils.equalsIgnoreCase(DatabaseIdentifier.KINGBASE, sourceCatalogName)
+                        ? column.getSourceType()
+                        : KingbaseTypeConverter.INSTANCE.reconvert(column).getColumnType();
+        columnSql.append(columnType);
+
+        if (!column.isNullable()) {
+            columnSql.append(" NOT NULL");
+        }
+
+        return columnSql.toString();
+    }
+
+    private String buildPrimaryKeySql(PrimaryKey primaryKey) {
+        String randomSuffix = UUID.randomUUID().toString().replace("-", "").substring(0, 4);
+        String columnNamesString =
+                primaryKey.getColumnNames().stream()
+                        .map(columnName -> "\"" + columnName + "\"")
+                        .collect(Collectors.joining(", "));
+
+        String primaryKeyStr = primaryKey.getPrimaryKey();
+        if (primaryKeyStr.length() > 25) {
+            primaryKeyStr = primaryKeyStr.substring(0, 25);
+        }
+
+        return CatalogUtils.getFieldIde(
+                "CONSTRAINT "
+                        + primaryKeyStr
+                        + "_"
+                        + randomSuffix
+                        + " PRIMARY KEY ("
+                        + columnNamesString
+                        + ")",
+                fieldIde);
+    }
+
+    private String buildColumnCommentSql(Column column, String tableName) {
+        StringBuilder columnCommentSql = new StringBuilder();
+        columnCommentSql
+                .append(CatalogUtils.quoteIdentifier("COMMENT ON COLUMN ", fieldIde))
+                .append(tableName)
+                .append(".");
+        columnCommentSql
+                .append(CatalogUtils.quoteIdentifier(column.getName(), fieldIde, "\""))
+                .append(CatalogUtils.quoteIdentifier(" IS '", fieldIde))
+                .append(column.getComment())
+                .append("'");

Review Comment:
   Thank you for your review. Please note that this feature is already available in other connectors. I forgot to add it and will include it later.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Featrue/Add Kingbase Catalog Support [seatunnel]

Posted by "LeonYoah (via GitHub)" <gi...@apache.org>.
LeonYoah commented on PR #10427:
URL: https://github.com/apache/seatunnel/pull/10427#issuecomment-3828600383

   > Hi @LeonYoah, thanks for this contribution! Adding Catalog support for Kingbase is a great enhancement.
   > 
   > After reviewing the code, I have a few suggestions to improve code consistency and compatibility:
   > 
   > ## 1. Naming Inconsistency
   > The existing classes in the project typically use `Kingbase` (only 'K' capitalized), such as `KingbaseDialect` 和 `KingbaseTypeMapper`。 However, the new classes introduced in this PR use `KingBase` (CamelCase 'K' and 'B'), such as `KingBaseCatalog` 和 `KingBaseCatalogFactory`。
   > 
   > **Suggestion:** Please rename the new classes to match the existing convention:
   > 
   > * `KingBaseCatalog` -> `KingbaseCatalog`
   > * `KingBaseCatalogFactory` -> `KingbaseCatalogFactory`
   > * `KingBaseCreateTableSqlBuilder` -> `KingbaseCreateTableSqlBuilder`
   > 
   > ## 2. Compatibility Risk with `rownum`
   > In `KingBaseCatalog.java`:
   > 
   > ```java
   > @Override
   > protected String getExistDataSql(TablePath tablePath) {
   >     return String.format(
   >             "select * from \"%s\".\"%s\" WHERE rownum = 1",
   >             tablePath.getSchemaName(), tablePath.getTableName());
   > }
   > ```
   > 
   > **Risk:** `rownum` is an Oracle-specific syntax. While Kingbase supports an Oracle compatibility mode, it usually runs in Postgres compatibility mode by default, where `rownum` might not be supported or behaves differently. This could cause the catalog to fail in environments running in standard PG mode.
   > 
   > **Suggestion:** Unless `rownum` is guaranteed to work in all Kingbase modes, it is safer to use the standard SQL standard or Postgres syntax: `LIMIT 1`。
   > 
   > ```java
   > "select * from \"%s\".\"%s\" LIMIT 1"
   > ```
   > 
   > ## 3. Documentation
   > It would be helpful to update the `Features` section in `docs/en/connectors/source/Kingbase.md` to indicate that Kingbase now supports the Catalog feature.
   
   Thank you for your review. I will make the changes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature]Add Kingbase Catalog Support [seatunnel]

Posted by "LeonYoah (via GitHub)" <gi...@apache.org>.
LeonYoah commented on code in PR #10427:
URL: https://github.com/apache/seatunnel/pull/10427#discussion_r2752358128


##########
seatunnel-connectors-v2/connector-jdbc/src/test/java/org/apache/seatunnel/connectors/seatunnel/jdbc/catalog/kingbase/KingbaseCatalogTest.java:
##########
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.seatunnel.connectors.seatunnel.jdbc.catalog.kingbase;
+
+import org.apache.seatunnel.shade.com.google.common.collect.Lists;
+
+import org.apache.seatunnel.api.table.catalog.CatalogTable;
+import org.apache.seatunnel.api.table.catalog.TablePath;
+import org.apache.seatunnel.common.utils.JdbcUrlUtil;
+
+import org.junit.jupiter.api.AfterAll;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.Disabled;
+import org.junit.jupiter.api.Test;
+
+import java.util.List;
+
+@Disabled("Please Test it in your local environment")

Review Comment:
   OK, I will modify the Kingbase unit test support and add container test support.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org