You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "chl-wxp (via GitHub)" <gi...@apache.org> on 2026/01/27 02:01:15 UTC

[PR] [Feature][seatunnel-api] Integrate Gravitino as metadata service for non-relational connectors [seatunnel]

chl-wxp opened a new pull request, #10402:
URL: https://github.com/apache/seatunnel/pull/10402

    ### What this PR does                                                                                                                                                                                                            
                                                                                                                                                                                                                                      
     Integrates Apache Gravitino as an external metadata service for non-relational connectors (Elasticsearch, vector databases, data lakes, etc.). Users can now query table schemas via Gravitino REST API instead of manually      
     defining complex schema mappings.                                                                                                                                                                                                
                                                                                                                                                                                                                                      
     ### Key Changes                                                                                                                                                                                                                  
                                                                                                                                                                                                                                      
     - Added `MetaLakeFactory` for creating metadata service clients                                                                                                                                                                  
     - Added `TableSchemaDiscoverer` interface for schema discovery                                                                                                                                                                   
     - Implemented `GravitinoClient` and `GravitinoTableSchemaConvertor`                                                                                                                                                              
     - Supports all Gravitino column types: boolean, integer, long, float, double, string, decimal, date, time, timestamp, binary, list, map, etc.                                                                                    
     - Schema validation is performed on the client side before job submission                                                                                                                                                        
                                                                                                                                                                                                                                      
     ### Benefits                                                                                                                                                                                                                     
                                                                                                                                                                                                                                      
     - No manual schema configuration required                                                                                                                                                                                        
     - Single source of truth for table schemas                                                                                                                                                                                       
     - Better consistency between metadata and actual storage                                                                                                                                                                         
                                                                                                                                                                                                                                      
     ### Related Issue                                                                                                                                                                                                                
                                                                                                                                                                                                                                      
     Fixes #10339                                                                                                                                                                                                                     
                                                                                                                                                                                                                                      


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][seatunnel-api] Integrate Gravitino as metadata service for non-relational connectors [seatunnel]

Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #10402:
URL: https://github.com/apache/seatunnel/pull/10402#discussion_r2791765007


##########
seatunnel-api/src/main/java/org/apache/seatunnel/api/metalake/TableSchemaDiscoverer.java:
##########
@@ -0,0 +1,239 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.seatunnel.api.metalake;
+
+import org.apache.seatunnel.shade.com.fasterxml.jackson.databind.JsonNode;
+import org.apache.seatunnel.shade.com.google.common.annotations.VisibleForTesting;
+import org.apache.seatunnel.shade.org.apache.commons.lang3.StringUtils;
+
+import org.apache.seatunnel.api.configuration.ReadonlyConfig;
+import org.apache.seatunnel.api.options.ConnectorCommonOptions;
+import org.apache.seatunnel.api.options.EnvCommonOptions;
+import org.apache.seatunnel.api.options.table.CatalogOptions;
+import org.apache.seatunnel.api.options.table.ColumnOptions;
+import org.apache.seatunnel.api.options.table.FieldOptions;
+import org.apache.seatunnel.api.options.table.TableIdentifierOptions;
+import org.apache.seatunnel.api.options.table.TableSchemaOptions;
+import org.apache.seatunnel.api.table.catalog.CatalogTable;
+import org.apache.seatunnel.api.table.catalog.CatalogTableUtil;
+import org.apache.seatunnel.api.table.catalog.TableIdentifier;
+import org.apache.seatunnel.api.table.catalog.TablePath;
+import org.apache.seatunnel.api.table.catalog.TableSchema;
+import org.apache.seatunnel.api.table.factory.TableSourceFactoryContext;
+import org.apache.seatunnel.common.constants.MetaLakeType;
+import org.apache.seatunnel.common.exception.SeaTunnelRuntimeException;
+
+import lombok.extern.slf4j.Slf4j;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.function.Predicate;
+import java.util.stream.Collectors;
+
+import static org.apache.seatunnel.api.table.schema.exception.SchemaEvolutionErrorCode.GET_META_LAKE_TABLE_SCHEMA_FAILED;
+
+@Slf4j
+public class TableSchemaDiscoverer implements AutoCloseable {
+
+    private final ReadonlyConfig envOptions;
+    private final ReadonlyConfig sourceOptions;
+    private final String catalogName;
+    private MetalakeClient metalakeClient;
+    private final MetaLakeTableSchemaConvertor metaLakeTableSchemaConvertor;
+
+    public TableSchemaDiscoverer(TableSourceFactoryContext context, String catalogName) {
+        this.envOptions = context.getEnvOptions();
+        this.sourceOptions = context.getOptions();
+        this.catalogName = catalogName;
+        if (enableMetaLakeClient(context.getOptions())) {
+            this.metalakeClient = MetaLakeFactory.createClient(getMetaLakeType());
+        }
+        this.metaLakeTableSchemaConvertor = MetaLakeFactory.createTypeMapper(getMetaLakeType());
+    }
+
+    @VisibleForTesting
+    protected TableSchemaDiscoverer(
+            ReadonlyConfig envOptions,
+            ReadonlyConfig sourceOptions,
+            String catalogName,
+            MetalakeClient metalakeClient,
+            MetaLakeTableSchemaConvertor convertor) {
+        this.envOptions = envOptions;
+        this.sourceOptions = sourceOptions;
+        this.catalogName = catalogName;
+        this.metalakeClient = metalakeClient;
+        this.metaLakeTableSchemaConvertor = convertor;
+    }
+
+    public List<CatalogTable> discoverTableSchemas() {
+        // schema
+        if (sourceOptions.getOptional(ConnectorCommonOptions.SCHEMA).isPresent()) {
+            return Collections.singletonList(discoverTableSchema(sourceOptions));
+        }
+        // table_config
+        if (sourceOptions.getOptional(TableSchemaOptions.TABLE_CONFIGS).isPresent()) {
+            return sourceOptions.get(TableSchemaOptions.TABLE_CONFIGS).stream()
+                    .map(ReadonlyConfig::fromMap)
+                    .map(this::discoverTableSchema)
+                    .collect(Collectors.toList());
+        }
+        // table_list
+        if (sourceOptions.getOptional(CatalogOptions.TABLE_LIST).isPresent()) {
+            return sourceOptions.get(CatalogOptions.TABLE_LIST).stream()
+                    .map(ReadonlyConfig::fromMap)
+                    .map(this::discoverTableSchema)
+                    .collect(Collectors.toList());
+        }
+        return Collections.singletonList(CatalogTableUtil.buildSimpleTextTable());
+    }
+
+    private CatalogTable discoverTableSchema(ReadonlyConfig sourceOptions) {
+        final Map<String, Object> schemaMap = sourceOptions.get(ConnectorCommonOptions.SCHEMA);
+        ReadonlyConfig schemaConfig = ReadonlyConfig.fromMap(schemaMap);
+        // fields or columns
+        if (schemaConfig.getOptional(ColumnOptions.COLUMNS).isPresent()
+                || sourceOptions.getOptional(FieldOptions.FIELDS).isPresent()) {
+            return discoverTableSchemaFromConfig(sourceOptions);
+        }

Review Comment:
   `discoverTableSchema` unconditionally reads `ConnectorCommonOptions.SCHEMA` from `sourceOptions` and then checks `sourceOptions.getOptional(FieldOptions.FIELDS)`. In the provided configs/tests, `fields` lives inside the `schema { ... }` block, so this condition will never trigger and the method will incorrectly fall back to `buildSimpleTextTable()`.
   
   Consider (1) handling the case where `schema` is absent (return simple text table or config-derived table), and (2) checking `schemaConfig.getOptional(FieldOptions.FIELDS)` (and/or `schemaConfig.getOptional(ColumnOptions.COLUMNS)`) instead of looking at `sourceOptions` for `FIELDS`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][seatunnel-api] Integrate Gravitino as metadata service for non-relational connectors [seatunnel]

Posted by "liugddx (via GitHub)" <gi...@apache.org>.
liugddx commented on code in PR #10402:
URL: https://github.com/apache/seatunnel/pull/10402#discussion_r2778671297


##########
seatunnel-api/src/main/java/org/apache/seatunnel/api/metalake/gravitino/GravitinoClient.java:
##########
@@ -0,0 +1,229 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.seatunnel.api.metalake.gravitino;
+
+import org.apache.seatunnel.shade.com.fasterxml.jackson.databind.JsonNode;
+import org.apache.seatunnel.shade.com.google.common.annotations.VisibleForTesting;
+
+import org.apache.seatunnel.api.metalake.MetalakeClient;
+import org.apache.seatunnel.api.table.catalog.TablePath;
+import org.apache.seatunnel.common.exception.SeaTunnelRuntimeException;
+import org.apache.seatunnel.common.utils.JsonUtils;
+
+import org.apache.http.HttpEntity;
+import org.apache.http.client.methods.CloseableHttpResponse;
+import org.apache.http.client.methods.HttpGet;
+import org.apache.http.impl.client.CloseableHttpClient;
+import org.apache.http.impl.client.HttpClients;
+import org.apache.http.util.EntityUtils;
+
+import lombok.extern.slf4j.Slf4j;
+
+import java.io.IOException;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+import static org.apache.seatunnel.api.table.schema.exception.SchemaEvolutionErrorCode.ERROR_INVALID_TABLE_URL;
+
+@Slf4j
+public class GravitinoClient implements MetalakeClient {
+
+    private static final String HEADER_ACCEPT = "Accept";
+    private static final String MEDIA_TYPE_GRAVITINO_V1 = "application/vnd.gravitino.v1+json";
+    private static final String JSON_FIELD_CATALOG = "catalog";
+    private static final String JSON_FIELD_TABLE = "table";
+    private static final String JSON_FIELD_PROPERTIES = "properties";
+    private static final String ERROR_NO_RESPONSE_ENTITY = "No response entity";
+    private static final String ERROR_MISSING_FIELD_TEMPLATE = "Response JSON has no '%s' field";
+    private static final int MAX_RETRY_ATTEMPTS = 3;
+    private static final long RETRY_DELAY_MS = 2000;
+    private static final Pattern TABLE_URL_PATTERN =
+            Pattern.compile("/catalogs/([^/]+)/schemas/([^/]+)/tables/([^/]+)");
+
+    private final CloseableHttpClient httpClient;
+
+    public GravitinoClient() {
+        this.httpClient = HttpClients.createDefault();
+    }
+
+    @VisibleForTesting
+    protected GravitinoClient(CloseableHttpClient httpClient) {
+        this.httpClient = httpClient;
+    }
+
+    @Override
+    public JsonNode getMetaInfo(String sourceId, String metalakeUrl) throws IOException {
+        JsonNode rootNode = executeGetRequest(metalakeUrl + sourceId);
+        JsonNode catalogNode = getRequiredNode(rootNode, JSON_FIELD_CATALOG);
+        return getRequiredNode(catalogNode, JSON_FIELD_PROPERTIES);
+    }
+
+    @Override
+    public JsonNode getTableSchema(String schemaHttpUrl) throws IOException {
+        JsonNode rootNode = executeGetRequest(schemaHttpUrl);
+        return getRequiredNode(rootNode, JSON_FIELD_TABLE);
+    }
+
+    @Override
+    public TablePath getTableSchemaPath(String schemaHttpUrl) {
+        if (schemaHttpUrl == null || schemaHttpUrl.isEmpty()) {
+            throw new SeaTunnelRuntimeException(
+                    ERROR_INVALID_TABLE_URL, "Table URL cannot be null or empty");
+        }
+        final Matcher matcher = getMatcher(schemaHttpUrl);
+        String catalogName = matcher.group(1);
+        String schemaName = matcher.group(2);
+        String tableName = matcher.group(3);
+        return TablePath.of(catalogName, schemaName, tableName);
+    }
+
+    private Matcher getMatcher(String schemaHttpUrl) {
+        Matcher matcher = TABLE_URL_PATTERN.matcher(schemaHttpUrl);
+        if (!matcher.find()) {
+            throw new SeaTunnelRuntimeException(
+                    ERROR_INVALID_TABLE_URL,
+                    String.format(
+                            "Invalid table URL format: '%s'. "
+                                    + "Expected format: http://host/api/metalakes/{metalake}/catalogs/{catalog}/schemas/{schema}/tables/{table}",
+                            schemaHttpUrl));
+        }
+        return matcher;
+    }
+
+    /**
+     * Execute HTTP GET request and return parsed JSON response. Implements retry with exponential
+     * backoff for transient failures.
+     *
+     * @param url the request URL
+     * @return parsed JSON root node
+     * @throws IOException if network or parsing error occurs after all retries
+     */
+    private JsonNode executeGetRequest(String url) throws IOException {
+        IOException lastException = null;
+        for (int attempt = 1; attempt <= MAX_RETRY_ATTEMPTS; attempt++) {
+            HttpGet request = new HttpGet(url);
+            request.addHeader(HEADER_ACCEPT, MEDIA_TYPE_GRAVITINO_V1);
+            try (CloseableHttpResponse response = httpClient.execute(request)) {
+                HttpEntity entity = response.getEntity();
+                if (entity == null) {
+                    throw new RuntimeException(ERROR_NO_RESPONSE_ENTITY);
+                }
+                try {
+                    return JsonUtils.readTree(entity.getContent());
+                } finally {
+                    EntityUtils.consume(entity);
+                }

Review Comment:
   The `response.getStatusLine().getStatusCode()` is not checked. If Gravitino returns a 4xx/5xx response, the code attempts to parse the error body as JSON, which may lead to misleading exceptions. Suggestion: Check the HTTP status code before reading the entity, and throw an exception containing the status code and body information for non-2xx responses.



##########
seatunnel-api/src/main/java/org/apache/seatunnel/api/table/factory/TableSourceFactory.java:
##########
@@ -40,6 +43,18 @@ TableSource<T, SplitT, StateT> createSource(TableSourceFactoryContext context) {
                 "The Factory has not been implemented and the deprecated Plugin will be used.");
     }
 
+    /**
+     * We can get the catalogTable list in the source configuration through this method
+     *
+     * @param context TableFactoryContext
+     */
+    default List<CatalogTable> discoverTableSchemas(TableSourceFactoryContext context) {
+        try (TableSchemaDiscoverer metaLakeSchemaDiscoverer =
+                new TableSchemaDiscoverer(context, factoryIdentifier())) {
+            return metaLakeSchemaDiscoverer.discoverTableSchemas();
+        }
+    }

Review Comment:
   All TableSourceFactory implementations will inherit this method by default, and even if the MetaLake feature is not used, a MetalakeClient (HTTP connection) will be created during construction. Suggestion: Implement lazy initialization in the TableSchemaDiscoverer constructor, creating the client only when there is indeed a schema_url configuration.



##########
seatunnel-api/src/main/java/org/apache/seatunnel/api/metalake/gravitino/GravitinoTableSchemaConvertor.java:
##########
@@ -0,0 +1,436 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.seatunnel.api.metalake.gravitino;
+
+import org.apache.seatunnel.shade.com.fasterxml.jackson.databind.JsonNode;
+
+import org.apache.seatunnel.api.metalake.MetaLakeTableSchemaConvertor;
+import org.apache.seatunnel.api.table.catalog.CatalogTable;
+import org.apache.seatunnel.api.table.catalog.Column;
+import org.apache.seatunnel.api.table.catalog.ConstraintKey;
+import org.apache.seatunnel.api.table.catalog.PhysicalColumn;
+import org.apache.seatunnel.api.table.catalog.PrimaryKey;
+import org.apache.seatunnel.api.table.catalog.TableIdentifier;
+import org.apache.seatunnel.api.table.catalog.TablePath;
+import org.apache.seatunnel.api.table.catalog.TableSchema;
+import org.apache.seatunnel.api.table.type.ArrayType;
+import org.apache.seatunnel.api.table.type.BasicType;
+import org.apache.seatunnel.api.table.type.DecimalType;
+import org.apache.seatunnel.api.table.type.LocalTimeType;
+import org.apache.seatunnel.api.table.type.MapType;
+import org.apache.seatunnel.api.table.type.PrimitiveByteArrayType;
+import org.apache.seatunnel.api.table.type.SeaTunnelDataType;
+import org.apache.seatunnel.api.table.type.SeaTunnelRowType;
+import org.apache.seatunnel.common.constants.MetaLakeType;
+import org.apache.seatunnel.common.exception.CommonError;
+
+import scala.Tuple2;

Review Comment:
   Introducing Scala dependencies in the pure Java seatunnel-api module may lead to unnecessary coupling. Recommendation: Use Java's built-in data structures (such as custom Pair classes or Map.Entry) as alternatives.



##########
seatunnel-api/src/main/java/org/apache/seatunnel/api/metalake/gravitino/GravitinoClient.java:
##########
@@ -0,0 +1,229 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.seatunnel.api.metalake.gravitino;
+
+import org.apache.seatunnel.shade.com.fasterxml.jackson.databind.JsonNode;
+import org.apache.seatunnel.shade.com.google.common.annotations.VisibleForTesting;
+
+import org.apache.seatunnel.api.metalake.MetalakeClient;
+import org.apache.seatunnel.api.table.catalog.TablePath;
+import org.apache.seatunnel.common.exception.SeaTunnelRuntimeException;
+import org.apache.seatunnel.common.utils.JsonUtils;
+
+import org.apache.http.HttpEntity;
+import org.apache.http.client.methods.CloseableHttpResponse;
+import org.apache.http.client.methods.HttpGet;
+import org.apache.http.impl.client.CloseableHttpClient;
+import org.apache.http.impl.client.HttpClients;
+import org.apache.http.util.EntityUtils;
+
+import lombok.extern.slf4j.Slf4j;
+
+import java.io.IOException;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+import static org.apache.seatunnel.api.table.schema.exception.SchemaEvolutionErrorCode.ERROR_INVALID_TABLE_URL;
+
+@Slf4j
+public class GravitinoClient implements MetalakeClient {
+
+    private static final String HEADER_ACCEPT = "Accept";
+    private static final String MEDIA_TYPE_GRAVITINO_V1 = "application/vnd.gravitino.v1+json";
+    private static final String JSON_FIELD_CATALOG = "catalog";
+    private static final String JSON_FIELD_TABLE = "table";
+    private static final String JSON_FIELD_PROPERTIES = "properties";
+    private static final String ERROR_NO_RESPONSE_ENTITY = "No response entity";
+    private static final String ERROR_MISSING_FIELD_TEMPLATE = "Response JSON has no '%s' field";
+    private static final int MAX_RETRY_ATTEMPTS = 3;
+    private static final long RETRY_DELAY_MS = 2000;
+    private static final Pattern TABLE_URL_PATTERN =
+            Pattern.compile("/catalogs/([^/]+)/schemas/([^/]+)/tables/([^/]+)");
+
+    private final CloseableHttpClient httpClient;
+
+    public GravitinoClient() {
+        this.httpClient = HttpClients.createDefault();
+    }
+
+    @VisibleForTesting
+    protected GravitinoClient(CloseableHttpClient httpClient) {
+        this.httpClient = httpClient;
+    }
+
+    @Override
+    public JsonNode getMetaInfo(String sourceId, String metalakeUrl) throws IOException {
+        JsonNode rootNode = executeGetRequest(metalakeUrl + sourceId);
+        JsonNode catalogNode = getRequiredNode(rootNode, JSON_FIELD_CATALOG);
+        return getRequiredNode(catalogNode, JSON_FIELD_PROPERTIES);
+    }
+
+    @Override
+    public JsonNode getTableSchema(String schemaHttpUrl) throws IOException {
+        JsonNode rootNode = executeGetRequest(schemaHttpUrl);
+        return getRequiredNode(rootNode, JSON_FIELD_TABLE);
+    }
+
+    @Override
+    public TablePath getTableSchemaPath(String schemaHttpUrl) {
+        if (schemaHttpUrl == null || schemaHttpUrl.isEmpty()) {
+            throw new SeaTunnelRuntimeException(
+                    ERROR_INVALID_TABLE_URL, "Table URL cannot be null or empty");
+        }
+        final Matcher matcher = getMatcher(schemaHttpUrl);
+        String catalogName = matcher.group(1);
+        String schemaName = matcher.group(2);
+        String tableName = matcher.group(3);
+        return TablePath.of(catalogName, schemaName, tableName);
+    }
+
+    private Matcher getMatcher(String schemaHttpUrl) {
+        Matcher matcher = TABLE_URL_PATTERN.matcher(schemaHttpUrl);
+        if (!matcher.find()) {
+            throw new SeaTunnelRuntimeException(
+                    ERROR_INVALID_TABLE_URL,
+                    String.format(
+                            "Invalid table URL format: '%s'. "
+                                    + "Expected format: http://host/api/metalakes/{metalake}/catalogs/{catalog}/schemas/{schema}/tables/{table}",
+                            schemaHttpUrl));
+        }
+        return matcher;
+    }
+
+    /**
+     * Execute HTTP GET request and return parsed JSON response. Implements retry with exponential
+     * backoff for transient failures.
+     *
+     * @param url the request URL
+     * @return parsed JSON root node
+     * @throws IOException if network or parsing error occurs after all retries
+     */
+    private JsonNode executeGetRequest(String url) throws IOException {
+        IOException lastException = null;
+        for (int attempt = 1; attempt <= MAX_RETRY_ATTEMPTS; attempt++) {
+            HttpGet request = new HttpGet(url);
+            request.addHeader(HEADER_ACCEPT, MEDIA_TYPE_GRAVITINO_V1);
+            try (CloseableHttpResponse response = httpClient.execute(request)) {
+                HttpEntity entity = response.getEntity();
+                if (entity == null) {
+                    throw new RuntimeException(ERROR_NO_RESPONSE_ENTITY);
+                }
+                try {
+                    return JsonUtils.readTree(entity.getContent());
+                } finally {
+                    EntityUtils.consume(entity);
+                }
+            } catch (IOException e) {
+                lastException = e;
+                // Check if exception is retryable
+                if (!isRetryableException(e) || attempt >= MAX_RETRY_ATTEMPTS) {
+                    break;
+                }
+                // Exponential backoff delay before retry
+                long delayMs = RETRY_DELAY_MS;
+                log.debug(
+                        "HTTP request to {} failed on attempt {}/{}, retrying in {}ms: {}",
+                        url,
+                        attempt,
+                        MAX_RETRY_ATTEMPTS,
+                        delayMs,
+                        e.getMessage());
+                sleepQuietly(delayMs);
+            }
+        }
+        throw new IOException(
+                String.format(
+                        "Failed to execute HTTP request to %s after %d attempts",
+                        url, MAX_RETRY_ATTEMPTS),
+                lastException);
+    }
+
+    /**
+     * Determine if an exception is retryable. Certain exceptions like DNS resolution failures, SSL
+     * errors, or 4xx client errors should not be retried as they will likely fail again.
+     *
+     * @param e the exception to check
+     * @return true if the exception is retryable, false otherwise
+     */
+    private boolean isRetryableException(IOException e) {
+        String message = e.getMessage();
+        if (message == null) {
+            return true;
+        }
+        // Non-retryable error patterns
+        String lowerMessage = message.toLowerCase();
+        if (lowerMessage.contains("unknownhost")
+                || lowerMessage.contains("dns")
+                || lowerMessage.contains("hostname")
+                || lowerMessage.contains("ssl")
+                || lowerMessage.contains("certificate")
+                || lowerMessage.contains("400")
+                || lowerMessage.contains("401")
+                || lowerMessage.contains("403")
+                || lowerMessage.contains("404")) {
+            log.debug("Non-retryable exception detected", e);
+            return false;
+        }

Review Comment:
   It is unlikely that the message of an IOException contains HTTP status code strings (Apache HttpClient's IOException is typically a connection/IO-level error). These checks will not actually take effect. The correct approach is to first check the response status code in executeGetRequest, throwing a non-retryable exception for 4xx errors and retrying for 5xx errors.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][seatunnel-api] Integrate Gravitino as metadata service for non-relational connectors [seatunnel]

Posted by "DanielCarter-stack (via GitHub)" <gi...@apache.org>.
DanielCarter-stack commented on PR #10402:
URL: https://github.com/apache/seatunnel/pull/10402#issuecomment-3853838337

   <!-- code-pr-reviewer -->
   <!-- cpr:pr_reply_v2_parts {"group": "apache/seatunnel#10402", "part": 1, "total": 1} -->
   ### Issue 1: HttpClient Resource Leak
   
   **Location**: `seatunnel-api/src/main/java/org/apache/seatunnel/api/metalake/gravitino/GravitinoClient.java:54-150`
   
   ```java
   private static volatile CloseableHttpClient httpClient;
   
   private CloseableHttpClient getHttpClient() {
       if (httpClient == null) {
           synchronized (GravitinoClient.class) {
               if (httpClient == null) {
                   httpClient = HttpClients.createDefault();
               }
           }
       }
       return httpClient;
   }
   ```
   
   **Related Context**:
   - Caller: `GravitinoClient.executeGetRequest()` (line 95)
   - HTTP request retry loop: lines 98-115
   - CloseableHttpResponse using try-with-resources: line 102
   
   **Issue Description**:
   `CloseableHttpClient` implements the `AutoCloseable` interface and manages connection pool and thread pool resources. The current code uses a singleton pattern but never calls the `close()` method, resulting in:
   1. Connections in the connection pool are never released
   2. Threads in the thread pool are never terminated
   3. In long-running processes, if the classloader reloads, multiple HttpClient instances will accumulate, exacerbating the resource leak
   
   **Potential Risks**:
   - **Risk 1**: File descriptor leak. On Linux systems, each TCP connection occupies a file descriptor. Long-running processes may lead to "Too many open files" errors.
   - **Risk 2**: Thread leak. HttpClient uses a connection pool by default and maintains threads internally. Not closing it causes thread accumulation.
   - **Risk 3**: Memory leak. Resources such as connection pools and DNS resolution caches will not be garbage collected.
   
   **Impact Scope**:
   - **Direct Impact**: All SeaTunnel jobs using `schema_url`
   - **Indirect Impact**: Long-running SeaTunnel Engine services
   - **Affected Area**: Core framework (seatunnel-api)
   
   **Severity**: BLOCKER
   
   **Improvement Suggestions**:
   
   ```java
   // Solution 1: Use shutdown hook (recommended)
   private static volatile CloseableHttpClient httpClient;
   private static volatile boolean closed = false;
   
   private CloseableHttpClient getHttpClient() {
       if (closed) {
           throw new IllegalStateException("HttpClient has been closed");
       }
       if (httpClient == null) {
           synchronized (GravitinoClient.class) {
               if (httpClient == null && !closed) {
                   httpClient = HttpClients.createDefault();
                   // Register JVM shutdown hook
                   Runtime.getRuntime().addShutdownHook(new Thread(() -> {
                       try {
                           if (httpClient != null) {
                               httpClient.close();
                           }
                       } catch (IOException e) {
                           // Ignore shutdown exceptions
                       }
                   }));
               }
           }
       }
       return httpClient;
   }
   
   // Solution 2: Do not use singleton, create new instance each time (simple but poor performance)
   private JsonNode executeGetRequest(String url) throws IOException {
       IOException lastException = null;
       for (int attempt = 1; attempt <= MAX_RETRY_ATTEMPTS; attempt++) {
           HttpGet request = new HttpGet(url);
           request.addHeader(HEADER_ACCEPT, MEDIA_TYPE_GRAVITINO_V1);
           
           try (CloseableHttpClient client = HttpClients.createDefault();
                CloseableHttpResponse response = client.execute(request)) {
               HttpEntity entity = response.getEntity();
               if (entity == null) {
                   throw new RuntimeException(ERROR_NO_RESPONSE_ENTITY);
               }
               try {
                   return JsonUtils.readTree(entity.getContent());
               } finally {
                   EntityUtils.consume(entity);
               }
           } catch (IOException e) {
               lastException = e;
           }
       }
       throw lastException;
   }
   
   // Solution 3: Implement HttpClient lifecycle management (most complete but complex)
   public class GravitinoClient implements MetalakeClient, AutoCloseable {
       private CloseableHttpClient httpClient;
       
       private synchronized CloseableHttpClient getHttpClient() {
           if (httpClient == null) {
               httpClient = HttpClients.createDefault();
           }
           return httpClient;
       }
       
       @Override
       public void close() throws IOException {
           if (httpClient != null) {
               httpClient.close();
               httpClient = null;
           }
       }
   }
   ```
   
   **Rationale**:
   - Option 1 balances performance (singleton reuse) and resource management (shutdown hook)
   - Option 2 is simple but has performance overhead (creating new client each time)
   - Option 3 is most complete but requires modifying caller code to ensure `close()` is called
   
   ---
   
   ### Issue 2: factoryIdentifier Misused as Catalog Name
   
   **Location**: `seatunnel-api/src/main/java/org/apache/seatunnel/api/table/factory/TableSourceFactory.java:48-52`
   
   ```java
   default List<CatalogTable> discoverTableSchemas(TableSourceFactoryContext context) {
       final TableSchemaDiscoverer metaLakeSchemaDiscoverer =
               new TableSchemaDiscoverer(context, factoryIdentifier());
       return metaLakeSchemaDiscoverer.discoverTableSchemas();
   }
   ```
   
   **Related Context**:
   - Interface definition: `TableSourceFactory.java:24` (`String factoryIdentifier()`)
   - Caller: `LocalFileSourceFactory.createSource()` (connector-file-local)
   - catalogName usage: `TableSchemaDiscoverer.java:122, 136, 147`
   
   **Issue Description**:
   `factoryIdentifier()` returns a connector identifier (such as "LocalFile", "HdfsFile"), which is not the semantic meaning of catalog. In `TableSchemaDiscoverer`, catalogName is used to:
   1. Construct the catalog name for `CatalogTable`
   2. As part of `TableIdentifier.of(catalogName, tablePath)`
   
   This results in:
   - When users configure `schema { table = "mydb.mytable" }`, the final catalog name will be replaced with "LocalFile"
   - Cannot correctly distinguish between different catalogs (such as production and test environments)
   
   **Potential Risks**:
   - **Risk 1**: Catalog name semantic confusion, making it difficult for users to understand why the catalog name became the connector name
   - **Risk 2**: If users specify a catalog name in the configuration (such as `table = "prod_db.table1"`), it will be overwritten
   
   **Impact Scope**:
   - **Direct Impact**: All scenarios using `schema_url` or `schema.fields`
   - **Indirect Impact**: Downstream functionality that relies on correct catalog names (such as multi-catalog writes)
   - **Affected Area**: All file connectors
   
   **Severity**: CRITICAL
   
   **Improvement Suggestions**:
   
   ```java
   // Solution 1: Parse catalog name from configuration (recommended)
   default List<CatalogTable> discoverTableSchemas(TableSourceFactoryContext context) {
       // Get catalog name from configuration, use factoryIdentifier as default if not available
       String catalogName = context.getOptions()
           .getOptional(ConnectorCommonOptions.CATALOG_NAME)  // Need to add this option
           .orElse(factoryIdentifier());
       
       final TableSchemaDiscoverer metaLakeSchemaDiscoverer =
               new TableSchemaDiscoverer(context, catalogName);
       return metaLakeSchemaDiscoverer.discoverTableSchemas();
   }
   
   // Solution 2: Parse catalog name from table path
   default List<CatalogTable> discoverTableSchemas(TableSourceFactoryContext context) {
       // Check the table property in configuration, extract catalog if it's a three-part name
       String catalogName = factoryIdentifier();
       
       Optional<String> configuredTable = context.getOptions()
           .getOptional(ConnectorCommonOptions.SCHEMA)
           .flatMap(schema -> schema.get("table"))
           .map(String::valueOf);
       
       if (configuredTable.isPresent()) {
           String tablePath = configuredTable.get();
           String[] parts = tablePath.split("\\.");
           if (parts.length == 3) {
               catalogName = parts[0];  // Use the configured catalog
           }
       }
       
       final TableSchemaDiscoverer metaLakeSchemaDiscoverer =
               new TableSchemaDiscoverer(context, catalogName);
       return metaLakeSchemaDiscoverer.discoverTableSchemas();
   }
   
   // Solution 3: Do not use factoryIdentifier, use fixed default value
   default List<CatalogTable> discoverTableSchemas(TableSourceFactoryContext context) {
       final TableSchemaDiscoverer metaLakeSchemaDiscoverer =
               new TableSchemaDiscoverer(context, "default");  // Use "default" as the default catalog
       return metaLakeSchemaDiscoverer.discoverTableSchemas();
   }
   ```
   
   **Rationale**:
   - Option 1 is most flexible, allowing users to customize catalog names
   - Option 2 has best compatibility, inferring catalog from existing configuration
   - Option 3 is simplest and aligns with most users' expectations (using "default" catalog)
   
   ---
   
   ### Issue 3: HTTP Retry Mechanism Lacks Backoff Strategy and Exception Classification
   
   **Location**: `seatunnel-api/src/main/java/org/apache/seatunnel/api/metalake/gravitino/GravitinoClient.java:95-118`
   
   ```java
   private JsonNode executeGetRequest(String url) throws IOException {
       IOException lastException = null;
   
       for (int attempt = 1; attempt <= MAX_RETRY_ATTEMPTS; attempt++) {
           HttpGet request = new HttpGet(url);
           request.addHeader(HEADER_ACCEPT, MEDIA_TYPE_GRAVITINO_V1);
   
           try (CloseableHttpResponse response = getHttpClient().execute(request)) {
               HttpEntity entity = response.getEntity();
               if (entity == null) {
                   throw new RuntimeException(ERROR_NO_RESPONSE_ENTITY);
               }
               try {
                   return JsonUtils.readTree(entity.getContent());
               } finally {
                   EntityUtils.consume(entity);
               }
           } catch (IOException e) {
               lastException = e;
           }
       }
   
       throw lastException;
   }
   ```
   
   **Related Context**:
   - Constant definitions: `MAX_RETRY_ATTEMPTS = 2` (line 50)
   - Caller: `getTableSchema()` (line 69)
   
   **Issue Description**:
   The current retry mechanism has the following issues:
   1. **No delay**: Retries are executed immediately, which may cause a thundering herd effect
   2. **No exception classification**: All IOExceptions are retried, but certain exceptions should not be retried (such as DNS resolution failures)
   3. **Missing exponential backoff**: Fixed number of retries, no delay adjustment based on failure count
   4. **Exception context loss**: The final thrown exception does not include URL and retry count information
   
   **Potential Risks**:
   - **Risk 1**: When the Gravitino service is temporarily unavailable, rapid retries may exacerbate service pressure
   - **Risk 2**: Exceptions such as DNS resolution failures and connection refusals should not be retried, but the current code will retry them
   - **Risk 3**: Difficult debugging, exception messages lack key information (such as URL, retry count)
   
   **Impact Scope**:
   - **Direct Impact**: All scenarios where metadata is obtained through `schema_url`
   - **Indirect Impact**: Stability of Gravitino service
   - **Affected Area**: Core framework
   
   **Severity**: MAJOR
   
   **Improvement Suggestions**:
   
   ```java
   private JsonNode executeGetRequest(String url) throws IOException {
       IOException lastException = null;
       long baseDelayMs = 1000;  // Base delay of 1 second
   
       for (int attempt = 1; attempt <= MAX_RETRY_ATTEMPTS; attempt++) {
           HttpGet request = new HttpGet(url);
           request.addHeader(HEADER_ACCEPT, MEDIA_TYPE_GRAVITINO_V1);
   
           try (CloseableHttpResponse response = getHttpClient().execute(request)) {
               HttpEntity entity = response.getEntity();
               if (entity == null) {
                   throw new RuntimeException(ERROR_NO_RESPONSE_ENTITY);
               }
               try {
                   return JsonUtils.readTree(entity.getContent());
               } finally {
                   EntityUtils.consume(entity);
               }
           } catch (IOException e) {
               lastException = e;
               
               // Determine if it should retry
               if (!shouldRetry(e) || attempt == MAX_RETRY_ATTEMPTS) {
                   break;
               }
               
               // Exponential backoff
               try {
                   long delayMs = baseDelayMs * (1L << (attempt - 1));
                   Thread.sleep(delayMs);
               } catch (InterruptedException ie) {
                   Thread.currentThread().interrupt();
                   throw new IOException("Retry interrupted", ie);
               }
           }
       }
   
       // Construct detailed exception information
       String errorMsg = String.format(
           "Failed to fetch schema from Gravitino after %d attempts. URL: %s. Last error: %s",
           MAX_RETRY_ATTEMPTS, url, lastException.getMessage()
       );
       throw new IOException(errorMsg, lastException);
   }
   
   /**
    * Determine if the exception should be retried
    */
   private boolean shouldRetry(IOException e) {
       // Non-retryable exception types
       if (e instanceof java.net.UnknownHostException) {
           return false;  // DNS resolution failed
       }
       if (e instanceof java.net.ConnectException) {
           return false;  // Connection refused
       }
       if (e instanceof org.apache.http.client.HttpResponseException) {
           int statusCode = ((org.apache.http.client.HttpResponseException) e).getStatusCode();
           if (statusCode == 404 || statusCode == 400 || statusCode == 401) {
               return false;  // Client errors should not be retried
           }
       }
       
       // Retryable exceptions: timeout, server errors (5xx)
       return true;
   }
   ```
   
   **Rationale**:
   - Exponential backoff prevents thundering herd effect
   - Exception classification avoids meaningless retries
   - Detailed exception information facilitates debugging
   - Aligns with best practices for network request retries
   
   ---
   
   ### Issue 4: Environment Variable Resolution Lacks Error Handling
   
   **Location**: `seatunnel-api/src/main/java/org/apache/seatunnel/api/metalake/TableSchemaDiscoverer.java:165-169`
   
   ```java
   if (StringUtils.isNotEmpty(
           System.getenv(EnvCommonOptions.METALAKE_TYPE.key().toUpperCase()))) {
       return MetaLakeType.valueOf(
               System.getenv(EnvCommonOptions.METALAKE_TYPE.key().toUpperCase()));
   }
   ```
   
   **Related Context**:
   - Enum definition: `MetaLakeType.java` (only one value: `GRAVITINO`)
   - Configuration option definition: `EnvCommonOptions.METALAKE_TYPE`
   
   **Issue Description**:
   When the environment variable `METALAKE_TYPE` is set to an invalid value (such as `invalid`), `MetaLakeType.valueOf()` throws `IllegalArgumentException`, causing job submission to fail. The error message is unfriendly, and users do not know what value to set.
   
   **Potential Risks**:
   - **Risk 1**: Users receive obscure error messages (`No enum constant MetaLakeType.invalid`) when configuration is incorrect
   - **Risk 2**: Lack of prompt information, users do not know the valid value is `gravitino`
   
   **Impact Scope**:
   - **Direct Impact**: Users using environment variable to configure `METALAKE_TYPE`
   - **Indirect Impact**: Difficult for operations personnel to troubleshoot
   - **Affected Area**: All scenarios using metadata service
   
   **Severity**: MINOR
   
   **Improvement Suggestions**:
   
   ```java
   // Solution 1: Use try-catch to provide friendly error messages
   if (StringUtils.isNotEmpty(
           System.getenv(EnvCommonOptions.METALAKE_TYPE.key().toUpperCase()))) {
       String metalakeTypeStr = System.getenv(EnvCommonOptions.METALAKE_TYPE.key().toUpperCase());
       try {
           return MetaLakeType.valueOf(metalakeTypeStr);
       } catch (IllegalArgumentException e) {
           throw new IllegalArgumentException(
               String.format(
                   "Invalid metalake type '%s'. Supported values are: %s. " +
                   "Please check your environment variable '%s' or configuration.",
                   metalakeTypeStr,
                   Arrays.stream(MetaLakeType.values())
                       .map(Enum::name)
                       .collect(Collectors.joining(", ")),
                   EnvCommonOptions.METALAKE_TYPE.key().toUpperCase()
               ),
               e
           );
       }
   }
   
   // Solution 2: Ignore invalid values, use default values and log warnings (more lenient)
   if (StringUtils.isNotEmpty(
           System.getenv(EnvCommonOptions.METALAKE_TYPE.key().toUpperCase()))) {
       String metalakeTypeStr = System.getenv(EnvCommonOptions.METALAKE_TYPE.key().toUpperCase());
       try {
           return MetaLakeType.valueOf(metalakeTypeStr);
       } catch (IllegalArgumentException e) {
           log.warn(
               "Invalid metalake type '{}' from environment variable '{}'. " +
               "Using default value '{}'. Supported values are: {}",
               metalakeTypeStr,
               EnvCommonOptions.METALAKE_TYPE.key().toUpperCase(),
               MetaLakeType.GRAVITINO.getType(),
               Arrays.stream(MetaLakeType.values())
                   .map(Enum::name)
                   .collect(Collectors.joining(", "))
           );
           return MetaLakeType.GRAVITINO;
       }
   }
   
   // Solution 3: Use case-insensitive matching (most robust)
   if (StringUtils.isNotEmpty(
           System.getenv(EnvCommonOptions.METALAKE_TYPE.key().toUpperCase()))) {
       String metalakeTypeStr = System.getenv(EnvCommonOptions.METALAKE_TYPE.key().toUpperCase());
       for (MetaLakeType type : MetaLakeType.values()) {
           if (type.name().equalsIgnoreCase(metalakeTypeStr)) {
               return type;
           }
       }
       // No matching type found
       throw new IllegalArgumentException(
           String.format(
               "Invalid metalake type '%s'. Supported values are: %s",
               metalakeTypeStr,
               Arrays.stream(MetaLakeType.values())
                   .map(Enum::name)
                   .collect(Collectors.joining(", "))
           )
       );
   }
   ```
   
   **Rationale**:
   - Option 1 provides most detailed error information, helping users quickly locate issues
   - Option 2 is most lenient but may hide configuration errors
   - Option 3 is most robust, supporting case-insensitive input
   
   ---
   
   ### Issue 5: Exception Handling Lacks Context Information
   
   **Location**: `seatunnel-api/src/main/java/org/apache/seatunnel/api/metalake/TableSchemaDiscoverer.java:125-140`
   
   ```java
   private CatalogTable discoverTableSchemaFromMetaLake(String schemaUrl, String configTablePath) {
       try {
           JsonNode schemaNode = metalakeClient.getTableSchema(schemaUrl);
           final TablePath tableSchemaPath;
           if (StringUtils.isNotEmpty(configTablePath)) {
               tableSchemaPath = TablePath.of(configTablePath);
           } else {
               tableSchemaPath = metalakeClient.getTableSchemaPath(schemaUrl);
           }
           final TableSchema tableSchema = metaLakeTableSchemaConvertor.convertor(schemaNode);
           return metaLakeTableSchemaConvertor.buildCatalogTable(
                   catalogName, tableSchemaPath, tableSchema);
       } catch (IOException e) {
           throw new SeaTunnelRuntimeException(GET_META_LAKE_TABLE_SCHEMA_FAILED, e);
       }
   }
   ```
   
   **Related Context**:
   - Error code definition: `SchemaEvolutionErrorCode.GET_META_LAKE_TABLE_SCHEMA_FAILED`
   - Exception message: `"Get meta lake table schema failed"`
   
   **Issue Description**:
   When fetching schema from Gravitino fails, the thrown `SeaTunnelRuntimeException` only contains the original `IOException`, lacking key context information:
   1. `schemaUrl` - User-configured URL
   2. `tablePath` - Parsed table path
   3. `catalogName` - Catalog name
   
   This makes debugging difficult, and users cannot quickly locate which table and which URL have the problem.
   
   **Potential Risks**:
   - **Risk 1**: In production environments, if multiple tables use `schema_url`, it is impossible to quickly locate which table failed from logs
   - **Risk 2**: Operations personnel need to manually check configuration files to determine the failed URL
   
   **Impact Scope**:
   - **Direct Impact**: All scenarios using `schema_url`
   - **Indirect Impact**: Troubleshooting efficiency
   - **Affected Area**: All file connectors
   
   **Severity**: MINOR
   
   **Improvement Suggestions**:
   
   ```java
   private CatalogTable discoverTableSchemaFromMetaLake(String schemaUrl, String configTablePath) {
       try {
           JsonNode schemaNode = metalakeClient.getTableSchema(schemaUrl);
           final TablePath tableSchemaPath;
           if (StringUtils.isNotEmpty(configTablePath)) {
               tableSchemaPath = TablePath.of(configTablePath);
           } else {
               tableSchemaPath = metalakeClient.getTableSchemaPath(schemaUrl);
           }
           final TableSchema tableSchema = metaLakeTableSchemaConvertor.convertor(schemaNode);
           return metaLakeTableSchemaConvertor.buildCatalogTable(
                   catalogName, tableSchemaPath, tableSchema);
       } catch (IOException e) {
           // Construct detailed error message
           String errorMsg = String.format(
               "Failed to get table schema from MetaLake. " +
               "Schema URL: %s, " +
               "Configured table path: %s, " +
               "Catalog name: %s, " +
               "Error: %s",
               schemaUrl,
               configTablePath != null ? configTablePath : "not configured",
               catalogName,
               e.getMessage()
           );
           throw new SeaTunnelRuntimeException(GET_META_LAKE_TABLE_SCHEMA_FAILED, 
               new IOException(errorMsg, e));
       }
   }
   ```
   
   **Rationale**:
   - Include all key context information in exception messages
   - Preserve original exception as cause for viewing complete stack trace
   - Aligns with exception handling best practices (fail-fast with context)
   
   ---
   
   ### Issue 6: BaseFileSourceConfig Constructor Signature Change Causes Breaking Change
   
   **Location**: `seatunnel-connectors-v2/connector-file/connector-file-base/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/config/BaseFileSourceConfig.java:57-60`
   
   ```java
   // Before
   public BaseFileSourceConfig(ReadonlyConfig readonlyConfig)
   
   // Now
   public BaseFileSourceConfig(ReadonlyConfig readonlyConfig, CatalogTable catalogTableFromConfig)
   ```
   
   **Related Context**:
   - Parent class: `BaseFileSourceConfig` (abstract class)
   - Subclasses: `BaseFileSourceConfig` implementations of 6 file connectors
   - Caller: `BaseMultipleTableFileSourceConfig`
   
   **Issue Description**:
   This is a breaking change that will cause all external code inheriting `BaseFileSourceConfig` to fail compilation. Although all internal subclasses in the current PR have been updated, there is no:
   1. Deprecation warning (`@Deprecated`)
   2. Migration guide
   3. Compatible transition solution
   
   **Potential Risks**:
   - **Risk 1**: If users have customized file connectors inheriting `BaseFileSourceConfig`, code will fail to compile
   - **Risk 2**: After upgrading to a new version, users need to modify code to compile successfully
   
   **Impact Scope**:
   - **Direct Impact**: All external code inheriting `BaseFileSourceConfig`
   - **Indirect Impact**: Upgrade compatibility
   - **Affected Area**: All file connectors
   
   **Severity**: MAJOR
   
   **Improvement Suggestions**:
   
   ```java
   // Solution 1: Keep the old constructor, mark as deprecated (recommended)
   @Deprecated
   public BaseFileSourceConfig(ReadonlyConfig readonlyConfig) {
       this(readonlyConfig, null);  // Call the new constructor
       log.warn(
           "BaseFileSourceConfig(ReadonlyConfig) is deprecated and will be removed in future versions. " +
           "Please use BaseFileSourceConfig(ReadonlyConfig, CatalogTable) instead."
       );
   }
   
   public BaseFileSourceConfig(ReadonlyConfig readonlyConfig, CatalogTable catalogTableFromConfig) {
       this.baseFileSourceConfig = readonlyConfig;
       this.fileFormat = readonlyConfig.get(FileBaseSourceOptions.FILE_FORMAT_TYPE);
       this.readStrategy = ReadStrategyFactory.of(readonlyConfig, getHadoopConfig());
       this.filePaths = parseFilePaths(readonlyConfig);
       this.catalogTableFromConfig = catalogTableFromConfig;
       this.catalogTable = parseCatalogTable(readonlyConfig);
   }
   
   // Solution 2: Use Builder pattern (more thorough improvement)
   public abstract class BaseFileSourceConfig implements Serializable {
       // ... field definitions
       
       protected BaseFileSourceConfig(Builder<?> builder) {
           this.baseFileSourceConfig = builder.readonlyConfig;
           this.fileFormat = builder.fileFormat;
           this.readStrategy = builder.readStrategy;
           this.filePaths = builder.filePaths;
           this.catalogTableFromConfig = builder.catalogTableFromConfig;
           this.catalogTable = parseCatalogTable(builder.readonlyConfig);
       }
       
       protected abstract static class Builder<T extends Builder<T>> {
           private ReadonlyConfig readonlyConfig;
           private FileFormat fileFormat;
           private ReadStrategy readStrategy;
           private List<String> filePaths;
           private CatalogTable catalogTableFromConfig;
           
           public T readonlyConfig(ReadonlyConfig readonlyConfig) {
               this.readonlyConfig = readonlyConfig;
               return self();
           }
           
           public T catalogTableFromConfig(CatalogTable catalogTableFromConfig) {
               this.catalogTableFromConfig = catalogTableFromConfig;
               return self();
           }
           
           protected abstract T self();
           
           public abstract BaseFileSourceConfig build();
       }
   }
   ```
   
   **Rationale**:
   - Option 1 provides backward compatibility, giving users time to migrate
   - Option 2 is more thorough, using Builder pattern to avoid the problem of too many constructor parameters
   
   ---
   
   ### Issue 7: GravitinoClient.getTableSchemaPath Regex Parsing Lacks Error Handling
   
   **Location**: `seatunnel-api/src/main/java/org/apache/seatunnel/api/metalake/gravitino/GravitinoClient.java:74-86`
   
   ```java
   @Override
   public TablePath getTableSchemaPath(String schemaHttpUrl) {
       Matcher matcher = TABLE_URL_PATTERN.matcher(schemaHttpUrl);
       if (!matcher.find()) {
           throw new SeaTunnelRuntimeException(
                   ERROR_INVALID_TABLE_URL,
                   "Invalid table URL format, expected: /catalogs/{catalog}/schemas/{schema}/tables/{table}");
       }
       String catalogName = matcher.group(1);
       String schemaName = matcher.group(2);
       String tableName = matcher.group(3);
       return TablePath.of(catalogName, schemaName, tableName);
   }
   ```
   
   **Related Context**:
   - Regex definition: `TABLE_URL_PATTERN = Pattern.compile("/catalogs/([^/]+)/schemas/([^/]+)/tables/([^/]+)")`
   - Error code: `ERROR_INVALID_TABLE_URL`
   
   **Issue Description**:
   When the URL format is incorrect, the thrown exception message lacks the actual URL, making it impossible for users to know which URL has the wrong format. Additionally, `matcher.find()` only finds the first match. If the URL contains multiple substrings that meet the condition, incorrect results may be parsed.
   
   **Potential Risks**:
   - **Risk 1**: Exception message does not contain the actual URL, making debugging difficult
   - **Risk 2**: `matcher.find()` may match the wrong location (although probability is very low)
   
   **Impact Scope**:
   - **Direct Impact**: All scenarios using `schema_url` without configuring `table` property
   - **Indirect Impact**: Troubleshooting efficiency
   - **Affected Area**: All file connectors
   
   **Severity**: MINOR
   
   **Improvement Suggestions**:
   
   ```java
   @Override
   public TablePath getTableSchemaPath(String schemaHttpUrl) {
       if (schemaHttpUrl == null || schemaHttpUrl.isEmpty()) {
           throw new SeaTunnelRuntimeException(
                   ERROR_INVALID_TABLE_URL,
                   "Table URL cannot be null or empty");
       }
       
       Matcher matcher = TABLE_URL_PATTERN.matcher(schemaHttpUrl);
       if (!matcher.matches()) {  // Use matches() instead of find()
           throw new SeaTunnelRuntimeException(
                   ERROR_INVALID_TABLE_URL,
                   String.format(
                       "Invalid table URL format: '%s'. " +
                       "Expected format: http://host/api/metalakes/{metalake}/catalogs/{catalog}/schemas/{schema}/tables/{table}",
                       schemaHttpUrl
                   ));
       }
       String catalogName = matcher.group(1);
       String schemaName = matcher.group(2);
       String tableName = matcher.group(3);
       return TablePath.of(catalogName, schemaName, tableName);
   }
   ```
   
   **Rationale**:
   - Use `matches()` to ensure the entire URL matches, not partial matching
   - Include the actual URL in exception messages
   - Add null checks to provide more friendly error messages
   
   ---
   
   ### Issue 8: HttpClient Connections May Become Invalid in Concurrent Scenarios
   
   **Location**: `seatunnel-api/src/main/java/org/apache/seatunnel/api/metalake/gravitino/GravitinoClient.java:141-150`
   
   ```java
   private CloseableHttpClient getHttpClient() {
       if (httpClient == null) {
           synchronized (GravitinoClient.class) {
               if (httpClient == null) {
                   httpClient = HttpClients.createDefault();
               }
           }
       }
       return httpClient;
   }
   ```
   
   **Issue Description**:
   Although `CloseableHttpClient` is thread-safe, the following scenarios may cause issues:
   1. After Gravitino service restarts, old connections in HttpClient may become invalid
   2. After DNS resolution changes, HttpClient may still use the old IP address
   3. In long-running processes, HttpClient's connection pool may encounter various issues (such as idle connections being closed by the server)
   
   Apache HttpClient's `createDefault()` method creates a client using the default connection manager, which automatically handles connection invalidation, but requires proper configuration.
   
   **Potential Risks**:
   - **Risk 1**: After Gravitino service restarts, requests in a short period may fail (until the connection manager detects connection invalidation)
   - **Risk 2**: Long-running processes may accumulate many invalid connections
   
   **Impact Scope**:
   - **Direct Impact**: Long-running SeaTunnel jobs
   - **Indirect Impact**: Gravitino service operations
   - **Affected Area**: All scenarios using `schema_url`
   
   **Severity**: MINOR
   
   **Improvement Suggestions**:
   
   ```java
   private CloseableHttpClient getHttpClient() {
       if (httpClient == null) {
           synchronized (GravitinoClient.class) {
               if (httpClient == null) {
                   // Configure connection manager to periodically check connection validity
                   PoolingHttpClientConnectionManager connectionManager =
                       PoolingHttpClientConnectionManagerBuilder.create()
                           .setMaxConnTotal(20)  // Maximum number of connections
                           .setMaxConnPerRoute(10)  // Maximum connections per route
                           .build();
                   
                   // Create HttpClient with reasonable timeout and retry configuration
                   RequestConfig requestConfig = RequestConfig.custom()
                       .setConnectTimeout(Timeout.ofMilliseconds(5000))  // Connection timeout of 5 seconds
                       .setConnectionRequestTimeout(Timeout.ofMilliseconds(5000))  // Timeout for getting connection from connection pool
                       .build();
                   
                   httpClient = HttpClients.custom()
                       .setConnectionManager(connectionManager)
                       .setDefaultRequestConfig(requestConfig)
                       .evictIdleConnections(30, TimeUnit.SECONDS)  // Clean up idle connections every 30 seconds
                       .build();
                       
                   // Register JVM shutdown hook
                   Runtime.getRuntime().addShutdownHook(new Thread(() -> {
                       try {
                           if (httpClient != null) {
                               httpClient.close();
                           }
                       } catch (IOException e) {
                           // Ignore shutdown exceptions
                       }
                   }));
               }
           }
       }
       return httpClient;
   }
   ```
   
   **Rationale**:
   - Configure connection pool parameters to avoid resource exhaustion
   - Periodically clean up idle connections to avoid using invalid connections
   - Set reasonable timeout values
   - Add shutdown hook to resolve resource leak issues
   
   ---


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][seatunnel-api] Integrate Gravitino as metadata service for non-relational connectors [seatunnel]

Posted by "chl-wxp (via GitHub)" <gi...@apache.org>.
chl-wxp commented on PR #10402:
URL: https://github.com/apache/seatunnel/pull/10402#issuecomment-3875702625

   In addition to file conflicts, please help and take a look again @liugddx  @zhangshenghang  @Carl-Zhou-CN @davidzollo @hawk9821 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][seatunnel-api] Integrate Gravitino as metadata service for non-relational connectors [seatunnel]

Posted by "hawk9821 (via GitHub)" <gi...@apache.org>.
hawk9821 commented on code in PR #10402:
URL: https://github.com/apache/seatunnel/pull/10402#discussion_r2739434456


##########
seatunnel-api/src/main/java/org/apache/seatunnel/api/metalake/gravitino/GravitinoTableSchemaConvertor.java:
##########
@@ -0,0 +1,345 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.seatunnel.api.metalake.gravitino;
+
+import org.apache.seatunnel.shade.com.fasterxml.jackson.databind.JsonNode;
+
+import org.apache.seatunnel.api.metalake.MetaLakeTableSchemaConvertor;
+import org.apache.seatunnel.api.table.catalog.CatalogTable;
+import org.apache.seatunnel.api.table.catalog.Column;
+import org.apache.seatunnel.api.table.catalog.ConstraintKey;
+import org.apache.seatunnel.api.table.catalog.PhysicalColumn;
+import org.apache.seatunnel.api.table.catalog.PrimaryKey;
+import org.apache.seatunnel.api.table.catalog.TableIdentifier;
+import org.apache.seatunnel.api.table.catalog.TablePath;
+import org.apache.seatunnel.api.table.catalog.TableSchema;
+import org.apache.seatunnel.api.table.type.ArrayType;
+import org.apache.seatunnel.api.table.type.BasicType;
+import org.apache.seatunnel.api.table.type.DecimalType;
+import org.apache.seatunnel.api.table.type.LocalTimeType;
+import org.apache.seatunnel.api.table.type.MapType;
+import org.apache.seatunnel.api.table.type.PrimitiveByteArrayType;
+import org.apache.seatunnel.api.table.type.SeaTunnelDataType;
+import org.apache.seatunnel.common.constants.MetaLakeType;
+import org.apache.seatunnel.common.exception.CommonError;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+/**
+ * Converter for transforming Gravitino table metadata into SeaTunnel CatalogTable format.
+ *
+ * <p>Reference documentation:
+ *
+ * <ul>
+ *   <li><a
+ *       href="https://gravitino.apache.org/docs/1.1.0/manage-relational-metadata-using-gravitino/#apache-gravitino-table-column-type">Gravitino
+ *       Column Types</a>
+ *   <li><a
+ *       href="https://gravitino.apache.org/docs/1.1.0/table-partitioning-distribution-sort-order-indexes#indexes">Gravitino
+ *       Indexes</a>
+ * </ul>
+ */
+public class GravitinoTableSchemaConvertor implements MetaLakeTableSchemaConvertor {
+
+    private static final Pattern DECIMAL_PATTERN =
+            Pattern.compile(
+                    "decimal\\s*\\(\\s*(\\d+)\\s*,\\s*(\\d+)\\s*\\)", Pattern.CASE_INSENSITIVE);
+    private static final Pattern VARCHAR_PATTERN =
+            Pattern.compile("varchar\\s*\\(\\s*(\\d+)\\s*\\)", Pattern.CASE_INSENSITIVE);
+    private static final Pattern CHAR_PATTERN =
+            Pattern.compile("char\\s*\\(\\s*(\\d+)\\s*\\)", Pattern.CASE_INSENSITIVE);
+    private static final Pattern FIXED_PATTERN =
+            Pattern.compile("fixed\\s*\\(\\s*(\\d+)\\s*\\)", Pattern.CASE_INSENSITIVE);
+
+    private static final String JSON_FIELD_COLUMNS = "columns";
+    private static final String JSON_FIELD_INDEXES = "indexes";
+    private static final String JSON_FIELD_NAME = "name";
+    private static final String JSON_FIELD_TYPE = "type";
+    private static final String JSON_FIELD_NULLABLE = "nullable";
+    private static final String JSON_FIELD_INDEX_TYPE = "indexType";
+    private static final String JSON_FIELD_FIELD_NAMES = "fieldNames";

Review Comment:
   `JSON_FIELD_FIELD_NAMES` has semantic redundancy
   



##########
seatunnel-api/src/main/java/org/apache/seatunnel/api/metalake/gravitino/GravitinoTableSchemaConvertor.java:
##########
@@ -0,0 +1,345 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.seatunnel.api.metalake.gravitino;
+
+import org.apache.seatunnel.shade.com.fasterxml.jackson.databind.JsonNode;
+
+import org.apache.seatunnel.api.metalake.MetaLakeTableSchemaConvertor;
+import org.apache.seatunnel.api.table.catalog.CatalogTable;
+import org.apache.seatunnel.api.table.catalog.Column;
+import org.apache.seatunnel.api.table.catalog.ConstraintKey;
+import org.apache.seatunnel.api.table.catalog.PhysicalColumn;
+import org.apache.seatunnel.api.table.catalog.PrimaryKey;
+import org.apache.seatunnel.api.table.catalog.TableIdentifier;
+import org.apache.seatunnel.api.table.catalog.TablePath;
+import org.apache.seatunnel.api.table.catalog.TableSchema;
+import org.apache.seatunnel.api.table.type.ArrayType;
+import org.apache.seatunnel.api.table.type.BasicType;
+import org.apache.seatunnel.api.table.type.DecimalType;
+import org.apache.seatunnel.api.table.type.LocalTimeType;
+import org.apache.seatunnel.api.table.type.MapType;
+import org.apache.seatunnel.api.table.type.PrimitiveByteArrayType;
+import org.apache.seatunnel.api.table.type.SeaTunnelDataType;
+import org.apache.seatunnel.common.constants.MetaLakeType;
+import org.apache.seatunnel.common.exception.CommonError;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+/**
+ * Converter for transforming Gravitino table metadata into SeaTunnel CatalogTable format.
+ *
+ * <p>Reference documentation:
+ *
+ * <ul>
+ *   <li><a
+ *       href="https://gravitino.apache.org/docs/1.1.0/manage-relational-metadata-using-gravitino/#apache-gravitino-table-column-type">Gravitino
+ *       Column Types</a>
+ *   <li><a
+ *       href="https://gravitino.apache.org/docs/1.1.0/table-partitioning-distribution-sort-order-indexes#indexes">Gravitino
+ *       Indexes</a>
+ * </ul>
+ */
+public class GravitinoTableSchemaConvertor implements MetaLakeTableSchemaConvertor {
+
+    private static final Pattern DECIMAL_PATTERN =
+            Pattern.compile(
+                    "decimal\\s*\\(\\s*(\\d+)\\s*,\\s*(\\d+)\\s*\\)", Pattern.CASE_INSENSITIVE);
+    private static final Pattern VARCHAR_PATTERN =
+            Pattern.compile("varchar\\s*\\(\\s*(\\d+)\\s*\\)", Pattern.CASE_INSENSITIVE);
+    private static final Pattern CHAR_PATTERN =
+            Pattern.compile("char\\s*\\(\\s*(\\d+)\\s*\\)", Pattern.CASE_INSENSITIVE);
+    private static final Pattern FIXED_PATTERN =
+            Pattern.compile("fixed\\s*\\(\\s*(\\d+)\\s*\\)", Pattern.CASE_INSENSITIVE);
+
+    private static final String JSON_FIELD_COLUMNS = "columns";
+    private static final String JSON_FIELD_INDEXES = "indexes";
+    private static final String JSON_FIELD_NAME = "name";
+    private static final String JSON_FIELD_TYPE = "type";
+    private static final String JSON_FIELD_NULLABLE = "nullable";
+    private static final String JSON_FIELD_INDEX_TYPE = "indexType";
+    private static final String JSON_FIELD_FIELD_NAMES = "fieldNames";
+
+    // Complex type JSON fields
+    private static final String JSON_FIELD_ELEMENT_TYPE = "elementType";
+    private static final String JSON_FIELD_KEY_TYPE = "keyType";
+    private static final String JSON_FIELD_VALUE_TYPE = "valueType";
+
+    private static final String INDEX_TYPE_PRIMARY_KEY = "PRIMARY_KEY";
+    private static final String INDEX_TYPE_UNIQUE_KEY = "UNIQUE_KEY";
+
+    @Override
+    public TableSchema convertor(JsonNode metaInfo) {
+        List<Column> columns = new ArrayList<>();
+        PrimaryKey primaryKey = null;
+        List<ConstraintKey> constraintKeys = new ArrayList<>();
+        // Parse columns
+        JsonNode columnsNode = metaInfo.get(JSON_FIELD_COLUMNS);
+        if (columnsNode != null && columnsNode.isArray()) {
+            for (JsonNode columnNode : columnsNode) {
+                columns.add(parseColumn(columnNode));
+            }
+        }
+        // Parse indexes
+        JsonNode indexesNode = metaInfo.get(JSON_FIELD_INDEXES);
+        if (indexesNode != null && indexesNode.isArray()) {
+            for (JsonNode indexNode : indexesNode) {
+                String indexType = getTextValue(indexNode, JSON_FIELD_INDEX_TYPE);
+                if (INDEX_TYPE_PRIMARY_KEY.equalsIgnoreCase(indexType)) {
+                    primaryKey = parsePrimaryKey(indexNode);
+                } else if (INDEX_TYPE_UNIQUE_KEY.equalsIgnoreCase(indexType)) {
+                    constraintKeys.add(parseUniqueKey(indexNode));
+                }
+            }
+        }
+        // Build table schema
+        TableSchema.Builder schemaBuilder = TableSchema.builder().columns(columns);
+        if (primaryKey != null) {
+            schemaBuilder.primaryKey(primaryKey);
+        }
+        if (!constraintKeys.isEmpty()) {
+            schemaBuilder.constraintKey(constraintKeys);
+        }
+        return schemaBuilder.build();
+    }
+
+    @Override
+    public CatalogTable buildCatalogTable(
+            String catalogName, TablePath tablePath, TableSchema tableSchema) {
+        TableIdentifier tableIdentifier = TableIdentifier.of(catalogName, tablePath);
+        // Build catalog table
+        return CatalogTable.of(
+                tableIdentifier,
+                tableSchema,
+                new HashMap<>(),
+                new ArrayList<>(),
+                null,
+                catalogName);
+    }
+
+    /** Parse a column node from Gravitino JSON. */
+    private Column parseColumn(JsonNode columnNode) {
+        String name = getTextValue(columnNode, JSON_FIELD_NAME);
+        boolean nullable =
+                columnNode.has(JSON_FIELD_NULLABLE)
+                        && columnNode.get(JSON_FIELD_NULLABLE).asBoolean();
+        JsonNode typeNode = columnNode.get(JSON_FIELD_TYPE);
+        if (typeNode == null) {
+            throw CommonError.convertToSeaTunnelTypeError(
+                    MetaLakeType.GRAVITINO.getType(), "null", name);
+        }
+        SeaTunnelDataType<?> dataType = convertGravitinoType(name, typeNode);
+        String typeStrForLength = typeNode.isTextual() ? typeNode.asText() : null;
+        Long columnLength = typeStrForLength == null ? null : extractColumnLength(typeStrForLength);
+        Integer scale = typeStrForLength == null ? null : extractScale(typeStrForLength);
+        return PhysicalColumn.builder()
+                .name(name)
+                .dataType(dataType)
+                .columnLength(columnLength)
+                .scale(scale)
+                .nullable(nullable)
+                .build();
+    }
+
+    /**
+     * Convert Gravitino type to SeaTunnel DataType.
+     *
+     * <p>Handles both simple types (string) and complex types (JSON object):
+     *
+     * <ul>
+     *   <li>Simple types: "integer", "varchar(100)", "decimal(10,2)", "map<string,int>",
+     *       "array<int>"
+     *   <li>Complex types:
+     *       <ul>
+     *         <li>list: {"type": "list", "containsNull": boolean, "elementType": type JSON}
+     *         <li>map: {"type": "map", "keyType": type JSON, "valueType": type JSON,
+     *             "valueContainsNull": boolean}
+     *       </ul>
+     * </ul>
+     *
+     * @param fieldName the field name for error reporting
+     * @param typeNode the JSON node representing the type (string or object)
+     * @return the corresponding SeaTunnel data type
+     */
+    private SeaTunnelDataType<?> convertGravitinoType(String fieldName, JsonNode typeNode) {
+        // Handle complex type (JSON object)
+        if (typeNode.isObject()) {
+            JsonNode typeField = typeNode.get(JSON_FIELD_TYPE);
+            if (typeField == null || !typeField.isTextual()) {
+                throw CommonError.convertToSeaTunnelTypeError(
+                        MetaLakeType.GRAVITINO.getType(), typeNode.toString(), fieldName);
+            }
+            String type = typeField.asText().toLowerCase();
+            switch (type) {
+                case "list":
+                    JsonNode elementType = typeNode.get(JSON_FIELD_ELEMENT_TYPE);
+                    if (elementType == null) {
+                        throw CommonError.convertToSeaTunnelTypeError(
+                                MetaLakeType.GRAVITINO.getType(),
+                                "list without elementType",
+                                fieldName);
+                    }
+                    return ArrayType.of(convertGravitinoType(fieldName, elementType));
+
+                case "map":
+                    JsonNode keyType = typeNode.get(JSON_FIELD_KEY_TYPE);
+                    JsonNode valueType = typeNode.get(JSON_FIELD_VALUE_TYPE);
+                    if (keyType == null || valueType == null) {
+                        throw CommonError.convertToSeaTunnelTypeError(
+                                MetaLakeType.GRAVITINO.getType(),
+                                "map without keyType or valueType",
+                                fieldName);
+                    }
+                    return new MapType<>(
+                            convertGravitinoType(fieldName, keyType),
+                            convertGravitinoType(fieldName, valueType));
+                case "struct":
+                case "union":
+                default:
+                    throw CommonError.convertToSeaTunnelTypeError(
+                            MetaLakeType.GRAVITINO.getType(), type, fieldName);
+            }
+        }
+        // Handle simple type (string)
+        if (!typeNode.isTextual()) {
+            throw CommonError.convertToSeaTunnelTypeError(
+                    MetaLakeType.GRAVITINO.getType(), typeNode.toString(), fieldName);
+        }
+        String gravitinoType = typeNode.asText();
+        String normalizedType = gravitinoType.trim().toLowerCase();
+        // Handle decimal type: decimal(precision, scale)
+        Matcher decimalMatcher = DECIMAL_PATTERN.matcher(gravitinoType);

Review Comment:
   The regular matching logic for DECIMAL_PATTERN is placed outside the switch statement. All simple types (including boolean, byte, string, and other types unrelated to decimal) will perform a regular matching operation before entering the type matching branch, resulting in a large amount of meaningless performance overhead. The higher the frequency of method calls, the more obvious the performance loss.



##########
seatunnel-api/src/main/java/org/apache/seatunnel/api/metalake/gravitino/GravitinoTableSchemaConvertor.java:
##########
@@ -0,0 +1,345 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.seatunnel.api.metalake.gravitino;
+
+import org.apache.seatunnel.shade.com.fasterxml.jackson.databind.JsonNode;
+
+import org.apache.seatunnel.api.metalake.MetaLakeTableSchemaConvertor;
+import org.apache.seatunnel.api.table.catalog.CatalogTable;
+import org.apache.seatunnel.api.table.catalog.Column;
+import org.apache.seatunnel.api.table.catalog.ConstraintKey;
+import org.apache.seatunnel.api.table.catalog.PhysicalColumn;
+import org.apache.seatunnel.api.table.catalog.PrimaryKey;
+import org.apache.seatunnel.api.table.catalog.TableIdentifier;
+import org.apache.seatunnel.api.table.catalog.TablePath;
+import org.apache.seatunnel.api.table.catalog.TableSchema;
+import org.apache.seatunnel.api.table.type.ArrayType;
+import org.apache.seatunnel.api.table.type.BasicType;
+import org.apache.seatunnel.api.table.type.DecimalType;
+import org.apache.seatunnel.api.table.type.LocalTimeType;
+import org.apache.seatunnel.api.table.type.MapType;
+import org.apache.seatunnel.api.table.type.PrimitiveByteArrayType;
+import org.apache.seatunnel.api.table.type.SeaTunnelDataType;
+import org.apache.seatunnel.common.constants.MetaLakeType;
+import org.apache.seatunnel.common.exception.CommonError;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+/**
+ * Converter for transforming Gravitino table metadata into SeaTunnel CatalogTable format.
+ *
+ * <p>Reference documentation:
+ *
+ * <ul>
+ *   <li><a
+ *       href="https://gravitino.apache.org/docs/1.1.0/manage-relational-metadata-using-gravitino/#apache-gravitino-table-column-type">Gravitino
+ *       Column Types</a>
+ *   <li><a
+ *       href="https://gravitino.apache.org/docs/1.1.0/table-partitioning-distribution-sort-order-indexes#indexes">Gravitino
+ *       Indexes</a>
+ * </ul>
+ */
+public class GravitinoTableSchemaConvertor implements MetaLakeTableSchemaConvertor {
+
+    private static final Pattern DECIMAL_PATTERN =
+            Pattern.compile(
+                    "decimal\\s*\\(\\s*(\\d+)\\s*,\\s*(\\d+)\\s*\\)", Pattern.CASE_INSENSITIVE);
+    private static final Pattern VARCHAR_PATTERN =
+            Pattern.compile("varchar\\s*\\(\\s*(\\d+)\\s*\\)", Pattern.CASE_INSENSITIVE);
+    private static final Pattern CHAR_PATTERN =
+            Pattern.compile("char\\s*\\(\\s*(\\d+)\\s*\\)", Pattern.CASE_INSENSITIVE);
+    private static final Pattern FIXED_PATTERN =
+            Pattern.compile("fixed\\s*\\(\\s*(\\d+)\\s*\\)", Pattern.CASE_INSENSITIVE);
+
+    private static final String JSON_FIELD_COLUMNS = "columns";
+    private static final String JSON_FIELD_INDEXES = "indexes";
+    private static final String JSON_FIELD_NAME = "name";
+    private static final String JSON_FIELD_TYPE = "type";
+    private static final String JSON_FIELD_NULLABLE = "nullable";
+    private static final String JSON_FIELD_INDEX_TYPE = "indexType";
+    private static final String JSON_FIELD_FIELD_NAMES = "fieldNames";
+
+    // Complex type JSON fields
+    private static final String JSON_FIELD_ELEMENT_TYPE = "elementType";
+    private static final String JSON_FIELD_KEY_TYPE = "keyType";
+    private static final String JSON_FIELD_VALUE_TYPE = "valueType";
+
+    private static final String INDEX_TYPE_PRIMARY_KEY = "PRIMARY_KEY";
+    private static final String INDEX_TYPE_UNIQUE_KEY = "UNIQUE_KEY";
+
+    @Override
+    public TableSchema convertor(JsonNode metaInfo) {
+        List<Column> columns = new ArrayList<>();
+        PrimaryKey primaryKey = null;
+        List<ConstraintKey> constraintKeys = new ArrayList<>();
+        // Parse columns
+        JsonNode columnsNode = metaInfo.get(JSON_FIELD_COLUMNS);
+        if (columnsNode != null && columnsNode.isArray()) {
+            for (JsonNode columnNode : columnsNode) {
+                columns.add(parseColumn(columnNode));
+            }
+        }
+        // Parse indexes
+        JsonNode indexesNode = metaInfo.get(JSON_FIELD_INDEXES);
+        if (indexesNode != null && indexesNode.isArray()) {
+            for (JsonNode indexNode : indexesNode) {
+                String indexType = getTextValue(indexNode, JSON_FIELD_INDEX_TYPE);
+                if (INDEX_TYPE_PRIMARY_KEY.equalsIgnoreCase(indexType)) {
+                    primaryKey = parsePrimaryKey(indexNode);
+                } else if (INDEX_TYPE_UNIQUE_KEY.equalsIgnoreCase(indexType)) {
+                    constraintKeys.add(parseUniqueKey(indexNode));
+                }
+            }
+        }
+        // Build table schema
+        TableSchema.Builder schemaBuilder = TableSchema.builder().columns(columns);
+        if (primaryKey != null) {
+            schemaBuilder.primaryKey(primaryKey);
+        }
+        if (!constraintKeys.isEmpty()) {
+            schemaBuilder.constraintKey(constraintKeys);
+        }
+        return schemaBuilder.build();
+    }
+
+    @Override
+    public CatalogTable buildCatalogTable(
+            String catalogName, TablePath tablePath, TableSchema tableSchema) {
+        TableIdentifier tableIdentifier = TableIdentifier.of(catalogName, tablePath);
+        // Build catalog table
+        return CatalogTable.of(
+                tableIdentifier,
+                tableSchema,
+                new HashMap<>(),
+                new ArrayList<>(),
+                null,
+                catalogName);
+    }
+
+    /** Parse a column node from Gravitino JSON. */
+    private Column parseColumn(JsonNode columnNode) {
+        String name = getTextValue(columnNode, JSON_FIELD_NAME);
+        boolean nullable =
+                columnNode.has(JSON_FIELD_NULLABLE)
+                        && columnNode.get(JSON_FIELD_NULLABLE).asBoolean();
+        JsonNode typeNode = columnNode.get(JSON_FIELD_TYPE);
+        if (typeNode == null) {
+            throw CommonError.convertToSeaTunnelTypeError(
+                    MetaLakeType.GRAVITINO.getType(), "null", name);
+        }
+        SeaTunnelDataType<?> dataType = convertGravitinoType(name, typeNode);
+        String typeStrForLength = typeNode.isTextual() ? typeNode.asText() : null;
+        Long columnLength = typeStrForLength == null ? null : extractColumnLength(typeStrForLength);
+        Integer scale = typeStrForLength == null ? null : extractScale(typeStrForLength);
+        return PhysicalColumn.builder()
+                .name(name)
+                .dataType(dataType)
+                .columnLength(columnLength)
+                .scale(scale)
+                .nullable(nullable)
+                .build();
+    }
+
+    /**
+     * Convert Gravitino type to SeaTunnel DataType.
+     *
+     * <p>Handles both simple types (string) and complex types (JSON object):
+     *
+     * <ul>
+     *   <li>Simple types: "integer", "varchar(100)", "decimal(10,2)", "map<string,int>",
+     *       "array<int>"
+     *   <li>Complex types:
+     *       <ul>
+     *         <li>list: {"type": "list", "containsNull": boolean, "elementType": type JSON}
+     *         <li>map: {"type": "map", "keyType": type JSON, "valueType": type JSON,
+     *             "valueContainsNull": boolean}
+     *       </ul>
+     * </ul>
+     *
+     * @param fieldName the field name for error reporting
+     * @param typeNode the JSON node representing the type (string or object)
+     * @return the corresponding SeaTunnel data type
+     */
+    private SeaTunnelDataType<?> convertGravitinoType(String fieldName, JsonNode typeNode) {
+        // Handle complex type (JSON object)
+        if (typeNode.isObject()) {
+            JsonNode typeField = typeNode.get(JSON_FIELD_TYPE);
+            if (typeField == null || !typeField.isTextual()) {
+                throw CommonError.convertToSeaTunnelTypeError(
+                        MetaLakeType.GRAVITINO.getType(), typeNode.toString(), fieldName);
+            }
+            String type = typeField.asText().toLowerCase();
+            switch (type) {
+                case "list":
+                    JsonNode elementType = typeNode.get(JSON_FIELD_ELEMENT_TYPE);
+                    if (elementType == null) {
+                        throw CommonError.convertToSeaTunnelTypeError(
+                                MetaLakeType.GRAVITINO.getType(),
+                                "list without elementType",
+                                fieldName);
+                    }
+                    return ArrayType.of(convertGravitinoType(fieldName, elementType));
+
+                case "map":
+                    JsonNode keyType = typeNode.get(JSON_FIELD_KEY_TYPE);
+                    JsonNode valueType = typeNode.get(JSON_FIELD_VALUE_TYPE);
+                    if (keyType == null || valueType == null) {
+                        throw CommonError.convertToSeaTunnelTypeError(
+                                MetaLakeType.GRAVITINO.getType(),
+                                "map without keyType or valueType",
+                                fieldName);
+                    }
+                    return new MapType<>(
+                            convertGravitinoType(fieldName, keyType),
+                            convertGravitinoType(fieldName, valueType));
+                case "struct":
+                case "union":
+                default:
+                    throw CommonError.convertToSeaTunnelTypeError(
+                            MetaLakeType.GRAVITINO.getType(), type, fieldName);
+            }
+        }
+        // Handle simple type (string)
+        if (!typeNode.isTextual()) {
+            throw CommonError.convertToSeaTunnelTypeError(
+                    MetaLakeType.GRAVITINO.getType(), typeNode.toString(), fieldName);
+        }
+        String gravitinoType = typeNode.asText();
+        String normalizedType = gravitinoType.trim().toLowerCase();
+        // Handle decimal type: decimal(precision, scale)
+        Matcher decimalMatcher = DECIMAL_PATTERN.matcher(gravitinoType);
+        if (decimalMatcher.find()) {
+            int precision = Integer.parseInt(decimalMatcher.group(1));
+            int scale = Integer.parseInt(decimalMatcher.group(2));
+            return new DecimalType(precision, scale);
+        }
+        // Remove parameters for simple type matching
+        String baseType = normalizedType.split("\\(")[0].trim();
+        switch (baseType) {
+            case "boolean":
+                return BasicType.BOOLEAN_TYPE;
+            case "byte":
+            case "byte unsigned":
+                return BasicType.BYTE_TYPE;
+            case "short":
+            case "short unsigned":
+                return BasicType.SHORT_TYPE;
+            case "integer":
+            case "integer unsigned":

Review Comment:
   byte unsigned, short unsigned, integer unsigned, and long unsigned are all matched through independent case branches. The mapping logic is completely consistent with that of their corresponding signed types, yet the branches are written repeatedly. When adding new unsigned types, additional cases need to be added, resulting in extremely low code scalability. We should preprocess the unsigned type to reduce switch branch logic.
   ```
   if (normalizedType.endsWith ("unsigned")) {
        baseType = normalizedType.replace ("unsigned", "").trim ();
   }
   
   ```



##########
seatunnel-api/src/main/java/org/apache/seatunnel/api/metalake/gravitino/GravitinoTableSchemaConvertor.java:
##########
@@ -0,0 +1,345 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.seatunnel.api.metalake.gravitino;
+
+import org.apache.seatunnel.shade.com.fasterxml.jackson.databind.JsonNode;
+
+import org.apache.seatunnel.api.metalake.MetaLakeTableSchemaConvertor;
+import org.apache.seatunnel.api.table.catalog.CatalogTable;
+import org.apache.seatunnel.api.table.catalog.Column;
+import org.apache.seatunnel.api.table.catalog.ConstraintKey;
+import org.apache.seatunnel.api.table.catalog.PhysicalColumn;
+import org.apache.seatunnel.api.table.catalog.PrimaryKey;
+import org.apache.seatunnel.api.table.catalog.TableIdentifier;
+import org.apache.seatunnel.api.table.catalog.TablePath;
+import org.apache.seatunnel.api.table.catalog.TableSchema;
+import org.apache.seatunnel.api.table.type.ArrayType;
+import org.apache.seatunnel.api.table.type.BasicType;
+import org.apache.seatunnel.api.table.type.DecimalType;
+import org.apache.seatunnel.api.table.type.LocalTimeType;
+import org.apache.seatunnel.api.table.type.MapType;
+import org.apache.seatunnel.api.table.type.PrimitiveByteArrayType;
+import org.apache.seatunnel.api.table.type.SeaTunnelDataType;
+import org.apache.seatunnel.common.constants.MetaLakeType;
+import org.apache.seatunnel.common.exception.CommonError;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+/**
+ * Converter for transforming Gravitino table metadata into SeaTunnel CatalogTable format.
+ *
+ * <p>Reference documentation:
+ *
+ * <ul>
+ *   <li><a
+ *       href="https://gravitino.apache.org/docs/1.1.0/manage-relational-metadata-using-gravitino/#apache-gravitino-table-column-type">Gravitino
+ *       Column Types</a>
+ *   <li><a
+ *       href="https://gravitino.apache.org/docs/1.1.0/table-partitioning-distribution-sort-order-indexes#indexes">Gravitino
+ *       Indexes</a>
+ * </ul>
+ */
+public class GravitinoTableSchemaConvertor implements MetaLakeTableSchemaConvertor {
+
+    private static final Pattern DECIMAL_PATTERN =
+            Pattern.compile(
+                    "decimal\\s*\\(\\s*(\\d+)\\s*,\\s*(\\d+)\\s*\\)", Pattern.CASE_INSENSITIVE);
+    private static final Pattern VARCHAR_PATTERN =
+            Pattern.compile("varchar\\s*\\(\\s*(\\d+)\\s*\\)", Pattern.CASE_INSENSITIVE);
+    private static final Pattern CHAR_PATTERN =
+            Pattern.compile("char\\s*\\(\\s*(\\d+)\\s*\\)", Pattern.CASE_INSENSITIVE);
+    private static final Pattern FIXED_PATTERN =
+            Pattern.compile("fixed\\s*\\(\\s*(\\d+)\\s*\\)", Pattern.CASE_INSENSITIVE);
+
+    private static final String JSON_FIELD_COLUMNS = "columns";
+    private static final String JSON_FIELD_INDEXES = "indexes";
+    private static final String JSON_FIELD_NAME = "name";
+    private static final String JSON_FIELD_TYPE = "type";
+    private static final String JSON_FIELD_NULLABLE = "nullable";
+    private static final String JSON_FIELD_INDEX_TYPE = "indexType";
+    private static final String JSON_FIELD_FIELD_NAMES = "fieldNames";
+
+    // Complex type JSON fields
+    private static final String JSON_FIELD_ELEMENT_TYPE = "elementType";
+    private static final String JSON_FIELD_KEY_TYPE = "keyType";
+    private static final String JSON_FIELD_VALUE_TYPE = "valueType";
+
+    private static final String INDEX_TYPE_PRIMARY_KEY = "PRIMARY_KEY";
+    private static final String INDEX_TYPE_UNIQUE_KEY = "UNIQUE_KEY";
+
+    @Override
+    public TableSchema convertor(JsonNode metaInfo) {
+        List<Column> columns = new ArrayList<>();
+        PrimaryKey primaryKey = null;
+        List<ConstraintKey> constraintKeys = new ArrayList<>();
+        // Parse columns
+        JsonNode columnsNode = metaInfo.get(JSON_FIELD_COLUMNS);
+        if (columnsNode != null && columnsNode.isArray()) {
+            for (JsonNode columnNode : columnsNode) {
+                columns.add(parseColumn(columnNode));
+            }
+        }
+        // Parse indexes
+        JsonNode indexesNode = metaInfo.get(JSON_FIELD_INDEXES);
+        if (indexesNode != null && indexesNode.isArray()) {
+            for (JsonNode indexNode : indexesNode) {
+                String indexType = getTextValue(indexNode, JSON_FIELD_INDEX_TYPE);
+                if (INDEX_TYPE_PRIMARY_KEY.equalsIgnoreCase(indexType)) {
+                    primaryKey = parsePrimaryKey(indexNode);
+                } else if (INDEX_TYPE_UNIQUE_KEY.equalsIgnoreCase(indexType)) {
+                    constraintKeys.add(parseUniqueKey(indexNode));
+                }
+            }
+        }
+        // Build table schema
+        TableSchema.Builder schemaBuilder = TableSchema.builder().columns(columns);
+        if (primaryKey != null) {
+            schemaBuilder.primaryKey(primaryKey);
+        }
+        if (!constraintKeys.isEmpty()) {
+            schemaBuilder.constraintKey(constraintKeys);
+        }
+        return schemaBuilder.build();
+    }
+
+    @Override
+    public CatalogTable buildCatalogTable(
+            String catalogName, TablePath tablePath, TableSchema tableSchema) {
+        TableIdentifier tableIdentifier = TableIdentifier.of(catalogName, tablePath);
+        // Build catalog table
+        return CatalogTable.of(
+                tableIdentifier,
+                tableSchema,
+                new HashMap<>(),
+                new ArrayList<>(),
+                null,
+                catalogName);
+    }
+
+    /** Parse a column node from Gravitino JSON. */
+    private Column parseColumn(JsonNode columnNode) {
+        String name = getTextValue(columnNode, JSON_FIELD_NAME);
+        boolean nullable =
+                columnNode.has(JSON_FIELD_NULLABLE)
+                        && columnNode.get(JSON_FIELD_NULLABLE).asBoolean();
+        JsonNode typeNode = columnNode.get(JSON_FIELD_TYPE);
+        if (typeNode == null) {
+            throw CommonError.convertToSeaTunnelTypeError(
+                    MetaLakeType.GRAVITINO.getType(), "null", name);
+        }
+        SeaTunnelDataType<?> dataType = convertGravitinoType(name, typeNode);
+        String typeStrForLength = typeNode.isTextual() ? typeNode.asText() : null;
+        Long columnLength = typeStrForLength == null ? null : extractColumnLength(typeStrForLength);
+        Integer scale = typeStrForLength == null ? null : extractScale(typeStrForLength);
+        return PhysicalColumn.builder()
+                .name(name)
+                .dataType(dataType)
+                .columnLength(columnLength)
+                .scale(scale)
+                .nullable(nullable)
+                .build();
+    }
+
+    /**
+     * Convert Gravitino type to SeaTunnel DataType.
+     *
+     * <p>Handles both simple types (string) and complex types (JSON object):
+     *
+     * <ul>
+     *   <li>Simple types: "integer", "varchar(100)", "decimal(10,2)", "map<string,int>",
+     *       "array<int>"
+     *   <li>Complex types:
+     *       <ul>
+     *         <li>list: {"type": "list", "containsNull": boolean, "elementType": type JSON}
+     *         <li>map: {"type": "map", "keyType": type JSON, "valueType": type JSON,
+     *             "valueContainsNull": boolean}
+     *       </ul>
+     * </ul>
+     *
+     * @param fieldName the field name for error reporting
+     * @param typeNode the JSON node representing the type (string or object)
+     * @return the corresponding SeaTunnel data type
+     */
+    private SeaTunnelDataType<?> convertGravitinoType(String fieldName, JsonNode typeNode) {
+        // Handle complex type (JSON object)
+        if (typeNode.isObject()) {

Review Comment:
   The code first handles complex types (Object) through an if statement. After processing, it does not break out and continues to execute the subsequent if judgment for simple types (!typeNode.isTextual()). Even if it is determined to be a complex type and processing is completed, it still performs irrelevant condition checks, resulting in invalid branch judgments and insufficient logical rigor.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][seatunnel-api] Integrate Gravitino as metadata service for non-relational connectors [seatunnel]

Posted by "hawk9821 (via GitHub)" <gi...@apache.org>.
hawk9821 commented on code in PR #10402:
URL: https://github.com/apache/seatunnel/pull/10402#discussion_r2791138836


##########
seatunnel-e2e/seatunnel-connector-v2-e2e/connector-file-local-e2e/src/test/java/org/apache/seatunnel/e2e/connector/file/local/LocalFileIT.java:
##########
@@ -70,9 +70,9 @@
 import java.util.zip.ZipOutputStream;
 
 @DisabledOnContainer(
-        value = {TestContainerId.SPARK_2_4},
-        type = {},
-        disabledReason = "The apache-compress version is not compatible with apache-poi")
+        value = {},
+        type = {EngineType.SPARK, EngineType.FLINK},

Review Comment:
   Why disable Spark and Flink use cases ?



##########
seatunnel-e2e/seatunnel-connector-v2-e2e/connector-file-local-e2e/src/test/java/org/apache/seatunnel/e2e/connector/file/local/LocalFileWithMetaLakeIT.java:
##########
@@ -0,0 +1,387 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *    Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.seatunnel.e2e.connector.file.local;
+
+import org.apache.seatunnel.e2e.common.container.ContainerExtendedFactory;
+import org.apache.seatunnel.e2e.common.container.seatunnel.SeaTunnelContainer;
+import org.apache.seatunnel.e2e.common.junit.TestContainerExtension;
+import org.apache.seatunnel.e2e.common.util.ContainerUtil;
+
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.testcontainers.containers.GenericContainer;
+import org.testcontainers.containers.MySQLContainer;
+import org.testcontainers.containers.output.Slf4jLogConsumer;
+import org.testcontainers.containers.wait.strategy.Wait;
+import org.testcontainers.images.PullPolicy;
+import org.testcontainers.utility.DockerImageName;
+import org.testcontainers.utility.DockerLoggerFactory;
+import org.testcontainers.utility.MountableFile;
+
+import lombok.extern.slf4j.Slf4j;
+
+import java.nio.file.Paths;
+import java.util.Arrays;
+import java.util.Collections;
+
+import static org.apache.seatunnel.e2e.common.util.ContainerUtil.PROJECT_ROOT_PATH;
+
+@Slf4j
+public class LocalFileWithMetaLakeIT extends SeaTunnelContainer {
+
+    private static final String GRAVITINO_IMAGE = "apache/gravitino:latest";
+    private static final int GRAVITINO_PORT = 8090;
+
+    private static final String MYSQL_IMAGE = "mysql:8.0.43";
+    private static final String MYSQL_CONTAINER_HOST = "mysql-e2e";
+    private static final String MYSQL_DATABASE = "seatunnel";
+    private static final String MYSQL_USERNAME = "root";
+    private static final String MYSQL_PASSWORD = "Abc!@#135_seatunnel";
+    private static final int MYSQL_PORT = 3306;
+
+    private GenericContainer<?> gravitinoContainer;
+    private GenericContainer<?> mysqlContainer;
+
+    @TestContainerExtension
+    private final ContainerExtendedFactory extendedFactory =
+            container -> {
+                // Copy CSV data files from resources to container
+                ContainerUtil.copyFileIntoContainers(
+                        "/csv/data/table1.csv",
+                        "/seatunnel/read/metalake/table1/data.csv",
+                        container);
+                ContainerUtil.copyFileIntoContainers(
+                        "/csv/data/table2.csv",
+                        "/seatunnel/read/metalake/table2/data.csv",
+                        container);
+            };
+
+    @BeforeEach
+    @Override
+    public void startUp() throws Exception {
+        // Start MySQL container first as metadata storage
+        startMySQLContainer();
+        // Start Gravitino server with MySQL as backend
+        startGravitinoServer();
+        // Start SeaTunnel server with MetaLake enabled
+        server =
+                new GenericContainer<>(getDockerImage())
+                        .withNetwork(NETWORK)
+                        .withEnv("TZ", "UTC")
+                        .withCommand(buildStartCommand())
+                        .withNetworkAliases("server")
+                        .withExposedPorts()
+                        .withFileSystemBind("/tmp", "/opt/hive")
+                        .withLogConsumer(
+                                new Slf4jLogConsumer(
+                                        DockerLoggerFactory.getLogger(
+                                                "seatunnel-engine:" + JDK_DOCKER_IMAGE)))
+                        .waitingFor(Wait.forLogMessage(".*received new worker register:.*", 1));
+        copySeaTunnelStarterToContainer(server);
+        server.setPortBindings(Arrays.asList("5801:5801", "8080:8080"));
+        server.withCopyFileToContainer(
+                MountableFile.forHostPath(
+                        PROJECT_ROOT_PATH
+                                + "/seatunnel-e2e/seatunnel-engine-e2e/connector-seatunnel-e2e-base/src/test/resources/"),
+                Paths.get(SEATUNNEL_HOME, "config").toString());
+
+        server.withCopyFileToContainer(
+                MountableFile.forHostPath(
+                        PROJECT_ROOT_PATH
+                                + "/seatunnel-shade/seatunnel-hadoop3-3.1.4-uber/target/seatunnel-hadoop3-3.1.4-uber.jar"),
+                Paths.get(SEATUNNEL_HOME, "lib/seatunnel-hadoop3-3.1.4-uber.jar").toString());
+
+        server.start();
+        // execute extra commands (including copying CSV files via extendedFactory)
+        // This must be called after server.start() because copyFileToContainer requires a running
+        // container
+        executeExtraCommands(extendedFactory);
+    }
+
+    @AfterEach
+    @Override
+    public void tearDown() throws Exception {
+        // Close containers in reverse order of creation
+        if (server != null) {
+            server.close();
+        }
+        if (gravitinoContainer != null) {
+            gravitinoContainer.close();
+        }
+        if (mysqlContainer != null) {
+            mysqlContainer.close();
+        }
+        // Note: Not calling super.tearDown() because:
+        // 1. This test overrides startUp() and doesn't use CONTAINER_VOLUME_MOUNT_PATH
+        // 2. Parent's tearDown tries to execInContainer on server which fails if already closed
+    }
+
+    private void startMySQLContainer() throws Exception {
+        DockerImageName imageName = DockerImageName.parse(MYSQL_IMAGE);
+        mysqlContainer =
+                new MySQLContainer<>(imageName)
+                        .withUsername(MYSQL_USERNAME)
+                        .withPassword(MYSQL_PASSWORD)
+                        .withDatabaseName(MYSQL_DATABASE)
+                        .withNetwork(NETWORK)
+                        .withNetworkAliases(MYSQL_CONTAINER_HOST)
+                        .withExposedPorts(MYSQL_PORT)
+                        .withImagePullPolicy(PullPolicy.alwaysPull())
+                        .waitingFor(Wait.forHealthcheck())
+                        .withLogConsumer(
+                                new Slf4jLogConsumer(DockerLoggerFactory.getLogger(MYSQL_IMAGE)));
+        mysqlContainer.setPortBindings(
+                Collections.singletonList(String.format("%s:%s", MYSQL_PORT, MYSQL_PORT)));
+        mysqlContainer.start();
+        log.info("MySQL container started at {}", mysqlContainer.getHost());
+        // Wait for MySQL to be fully ready
+        Thread.sleep(10000);
+    }
+
+    private void startGravitinoServer() throws Exception {
+        gravitinoContainer =
+                new GenericContainer<>(GRAVITINO_IMAGE)
+                        .withNetwork(NETWORK)
+                        .withNetworkAliases("gravitino")
+                        .withExposedPorts(GRAVITINO_PORT)
+                        .withLogConsumer(
+                                new Slf4jLogConsumer(
+                                        DockerLoggerFactory.getLogger(
+                                                "gravitino:" + GRAVITINO_IMAGE)));
+        gravitinoContainer.setPortBindings(
+                Collections.singletonList(String.format("%s:%s", GRAVITINO_PORT, GRAVITINO_PORT)));
+        gravitinoContainer.start();
+        // Wait for Gravitino to be ready by checking the API endpoint
+        waitForGravitinoReady();
+        log.info("Gravitino server started at {}", gravitinoContainer.getHost());
+        // Create metalake and catalog using curl with MySQL as backend
+        createMetalakeAndCatalog();
+    }
+
+    private void waitForGravitinoReady() throws Exception {
+        int maxAttempts = 60;
+        int attempt = 0;
+        while (attempt < maxAttempts) {

Review Comment:
   Can the stability of CI be guaranteed ?  



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][seatunnel-api] Integrate Gravitino as metadata service for non-relational connectors [seatunnel]

Posted by "hawk9821 (via GitHub)" <gi...@apache.org>.
hawk9821 commented on code in PR #10402:
URL: https://github.com/apache/seatunnel/pull/10402#discussion_r2791138836


##########
seatunnel-e2e/seatunnel-connector-v2-e2e/connector-file-local-e2e/src/test/java/org/apache/seatunnel/e2e/connector/file/local/LocalFileIT.java:
##########
@@ -70,9 +70,9 @@
 import java.util.zip.ZipOutputStream;
 
 @DisabledOnContainer(
-        value = {TestContainerId.SPARK_2_4},
-        type = {},
-        disabledReason = "The apache-compress version is not compatible with apache-poi")
+        value = {},
+        type = {EngineType.SPARK, EngineType.FLINK},

Review Comment:
   Why disable Spark and Flink e2e cases ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][seatunnel-api] Integrate Gravitino as metadata service for non-relational connectors [seatunnel]

Posted by "davidzollo (via GitHub)" <gi...@apache.org>.
davidzollo merged PR #10402:
URL: https://github.com/apache/seatunnel/pull/10402


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][seatunnel-api] Integrate Gravitino as metadata service for non-relational connectors [seatunnel]

Posted by "chl-wxp (via GitHub)" <gi...@apache.org>.
chl-wxp commented on PR #10402:
URL: https://github.com/apache/seatunnel/pull/10402#issuecomment-3858449992

   > Issue 6
   
   Issue 2 No changes, keep the previous usage.
   Issue 6 will not be changed. Relevant changes have been completed and no transition is required.
   Everything else has been changed.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][seatunnel-api] Integrate Gravitino as metadata service for non-relational connectors [seatunnel]

Posted by "hawk9821 (via GitHub)" <gi...@apache.org>.
hawk9821 commented on code in PR #10402:
URL: https://github.com/apache/seatunnel/pull/10402#discussion_r2739414182


##########
seatunnel-api/src/main/java/org/apache/seatunnel/api/metalake/gravitino/GravitinoTableSchemaConvertor.java:
##########
@@ -0,0 +1,345 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.seatunnel.api.metalake.gravitino;
+
+import org.apache.seatunnel.shade.com.fasterxml.jackson.databind.JsonNode;
+
+import org.apache.seatunnel.api.metalake.MetaLakeTableSchemaConvertor;
+import org.apache.seatunnel.api.table.catalog.CatalogTable;
+import org.apache.seatunnel.api.table.catalog.Column;
+import org.apache.seatunnel.api.table.catalog.ConstraintKey;
+import org.apache.seatunnel.api.table.catalog.PhysicalColumn;
+import org.apache.seatunnel.api.table.catalog.PrimaryKey;
+import org.apache.seatunnel.api.table.catalog.TableIdentifier;
+import org.apache.seatunnel.api.table.catalog.TablePath;
+import org.apache.seatunnel.api.table.catalog.TableSchema;
+import org.apache.seatunnel.api.table.type.ArrayType;
+import org.apache.seatunnel.api.table.type.BasicType;
+import org.apache.seatunnel.api.table.type.DecimalType;
+import org.apache.seatunnel.api.table.type.LocalTimeType;
+import org.apache.seatunnel.api.table.type.MapType;
+import org.apache.seatunnel.api.table.type.PrimitiveByteArrayType;
+import org.apache.seatunnel.api.table.type.SeaTunnelDataType;
+import org.apache.seatunnel.common.constants.MetaLakeType;
+import org.apache.seatunnel.common.exception.CommonError;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+/**
+ * Converter for transforming Gravitino table metadata into SeaTunnel CatalogTable format.
+ *
+ * <p>Reference documentation:
+ *
+ * <ul>
+ *   <li><a
+ *       href="https://gravitino.apache.org/docs/1.1.0/manage-relational-metadata-using-gravitino/#apache-gravitino-table-column-type">Gravitino
+ *       Column Types</a>
+ *   <li><a
+ *       href="https://gravitino.apache.org/docs/1.1.0/table-partitioning-distribution-sort-order-indexes#indexes">Gravitino
+ *       Indexes</a>
+ * </ul>
+ */
+public class GravitinoTableSchemaConvertor implements MetaLakeTableSchemaConvertor {
+
+    private static final Pattern DECIMAL_PATTERN =
+            Pattern.compile(
+                    "decimal\\s*\\(\\s*(\\d+)\\s*,\\s*(\\d+)\\s*\\)", Pattern.CASE_INSENSITIVE);
+    private static final Pattern VARCHAR_PATTERN =
+            Pattern.compile("varchar\\s*\\(\\s*(\\d+)\\s*\\)", Pattern.CASE_INSENSITIVE);
+    private static final Pattern CHAR_PATTERN =
+            Pattern.compile("char\\s*\\(\\s*(\\d+)\\s*\\)", Pattern.CASE_INSENSITIVE);
+    private static final Pattern FIXED_PATTERN =
+            Pattern.compile("fixed\\s*\\(\\s*(\\d+)\\s*\\)", Pattern.CASE_INSENSITIVE);
+
+    private static final String JSON_FIELD_COLUMNS = "columns";
+    private static final String JSON_FIELD_INDEXES = "indexes";
+    private static final String JSON_FIELD_NAME = "name";
+    private static final String JSON_FIELD_TYPE = "type";
+    private static final String JSON_FIELD_NULLABLE = "nullable";
+    private static final String JSON_FIELD_INDEX_TYPE = "indexType";
+    private static final String JSON_FIELD_FIELD_NAMES = "fieldNames";
+
+    // Complex type JSON fields
+    private static final String JSON_FIELD_ELEMENT_TYPE = "elementType";
+    private static final String JSON_FIELD_KEY_TYPE = "keyType";
+    private static final String JSON_FIELD_VALUE_TYPE = "valueType";
+
+    private static final String INDEX_TYPE_PRIMARY_KEY = "PRIMARY_KEY";
+    private static final String INDEX_TYPE_UNIQUE_KEY = "UNIQUE_KEY";
+
+    @Override
+    public TableSchema convertor(JsonNode metaInfo) {
+        List<Column> columns = new ArrayList<>();
+        PrimaryKey primaryKey = null;
+        List<ConstraintKey> constraintKeys = new ArrayList<>();
+        // Parse columns
+        JsonNode columnsNode = metaInfo.get(JSON_FIELD_COLUMNS);
+        if (columnsNode != null && columnsNode.isArray()) {
+            for (JsonNode columnNode : columnsNode) {
+                columns.add(parseColumn(columnNode));
+            }
+        }
+        // Parse indexes
+        JsonNode indexesNode = metaInfo.get(JSON_FIELD_INDEXES);
+        if (indexesNode != null && indexesNode.isArray()) {
+            for (JsonNode indexNode : indexesNode) {
+                String indexType = getTextValue(indexNode, JSON_FIELD_INDEX_TYPE);
+                if (INDEX_TYPE_PRIMARY_KEY.equalsIgnoreCase(indexType)) {
+                    primaryKey = parsePrimaryKey(indexNode);
+                } else if (INDEX_TYPE_UNIQUE_KEY.equalsIgnoreCase(indexType)) {
+                    constraintKeys.add(parseUniqueKey(indexNode));
+                }
+            }
+        }
+        // Build table schema
+        TableSchema.Builder schemaBuilder = TableSchema.builder().columns(columns);
+        if (primaryKey != null) {
+            schemaBuilder.primaryKey(primaryKey);
+        }
+        if (!constraintKeys.isEmpty()) {
+            schemaBuilder.constraintKey(constraintKeys);
+        }
+        return schemaBuilder.build();
+    }
+
+    @Override
+    public CatalogTable buildCatalogTable(
+            String catalogName, TablePath tablePath, TableSchema tableSchema) {
+        TableIdentifier tableIdentifier = TableIdentifier.of(catalogName, tablePath);
+        // Build catalog table
+        return CatalogTable.of(
+                tableIdentifier,
+                tableSchema,
+                new HashMap<>(),
+                new ArrayList<>(),
+                null,
+                catalogName);
+    }
+
+    /** Parse a column node from Gravitino JSON. */
+    private Column parseColumn(JsonNode columnNode) {
+        String name = getTextValue(columnNode, JSON_FIELD_NAME);
+        boolean nullable =
+                columnNode.has(JSON_FIELD_NULLABLE)
+                        && columnNode.get(JSON_FIELD_NULLABLE).asBoolean();
+        JsonNode typeNode = columnNode.get(JSON_FIELD_TYPE);
+        if (typeNode == null) {
+            throw CommonError.convertToSeaTunnelTypeError(
+                    MetaLakeType.GRAVITINO.getType(), "null", name);
+        }
+        SeaTunnelDataType<?> dataType = convertGravitinoType(name, typeNode);
+        String typeStrForLength = typeNode.isTextual() ? typeNode.asText() : null;
+        Long columnLength = typeStrForLength == null ? null : extractColumnLength(typeStrForLength);
+        Integer scale = typeStrForLength == null ? null : extractScale(typeStrForLength);
+        return PhysicalColumn.builder()
+                .name(name)
+                .dataType(dataType)
+                .columnLength(columnLength)
+                .scale(scale)
+                .nullable(nullable)
+                .build();
+    }
+
+    /**
+     * Convert Gravitino type to SeaTunnel DataType.
+     *
+     * <p>Handles both simple types (string) and complex types (JSON object):
+     *
+     * <ul>
+     *   <li>Simple types: "integer", "varchar(100)", "decimal(10,2)", "map<string,int>",
+     *       "array<int>"
+     *   <li>Complex types:
+     *       <ul>
+     *         <li>list: {"type": "list", "containsNull": boolean, "elementType": type JSON}
+     *         <li>map: {"type": "map", "keyType": type JSON, "valueType": type JSON,
+     *             "valueContainsNull": boolean}
+     *       </ul>
+     * </ul>
+     *
+     * @param fieldName the field name for error reporting
+     * @param typeNode the JSON node representing the type (string or object)
+     * @return the corresponding SeaTunnel data type
+     */
+    private SeaTunnelDataType<?> convertGravitinoType(String fieldName, JsonNode typeNode) {
+        // Handle complex type (JSON object)
+        if (typeNode.isObject()) {
+            JsonNode typeField = typeNode.get(JSON_FIELD_TYPE);
+            if (typeField == null || !typeField.isTextual()) {
+                throw CommonError.convertToSeaTunnelTypeError(
+                        MetaLakeType.GRAVITINO.getType(), typeNode.toString(), fieldName);
+            }
+            String type = typeField.asText().toLowerCase();
+            switch (type) {
+                case "list":
+                    JsonNode elementType = typeNode.get(JSON_FIELD_ELEMENT_TYPE);
+                    if (elementType == null) {
+                        throw CommonError.convertToSeaTunnelTypeError(
+                                MetaLakeType.GRAVITINO.getType(),
+                                "list without elementType",
+                                fieldName);
+                    }
+                    return ArrayType.of(convertGravitinoType(fieldName, elementType));
+
+                case "map":
+                    JsonNode keyType = typeNode.get(JSON_FIELD_KEY_TYPE);
+                    JsonNode valueType = typeNode.get(JSON_FIELD_VALUE_TYPE);
+                    if (keyType == null || valueType == null) {
+                        throw CommonError.convertToSeaTunnelTypeError(
+                                MetaLakeType.GRAVITINO.getType(),
+                                "map without keyType or valueType",
+                                fieldName);
+                    }
+                    return new MapType<>(
+                            convertGravitinoType(fieldName, keyType),
+                            convertGravitinoType(fieldName, valueType));
+                case "struct":
+                case "union":
+                default:
+                    throw CommonError.convertToSeaTunnelTypeError(
+                            MetaLakeType.GRAVITINO.getType(), type, fieldName);
+            }
+        }
+        // Handle simple type (string)
+        if (!typeNode.isTextual()) {
+            throw CommonError.convertToSeaTunnelTypeError(
+                    MetaLakeType.GRAVITINO.getType(), typeNode.toString(), fieldName);
+        }
+        String gravitinoType = typeNode.asText();
+        String normalizedType = gravitinoType.trim().toLowerCase();
+        // Handle decimal type: decimal(precision, scale)
+        Matcher decimalMatcher = DECIMAL_PATTERN.matcher(gravitinoType);
+        if (decimalMatcher.find()) {
+            int precision = Integer.parseInt(decimalMatcher.group(1));
+            int scale = Integer.parseInt(decimalMatcher.group(2));
+            return new DecimalType(precision, scale);
+        }
+        // Remove parameters for simple type matching
+        String baseType = normalizedType.split("\\(")[0].trim();
+        switch (baseType) {
+            case "boolean":
+                return BasicType.BOOLEAN_TYPE;
+            case "byte":
+            case "byte unsigned":
+                return BasicType.BYTE_TYPE;
+            case "short":
+            case "short unsigned":
+                return BasicType.SHORT_TYPE;
+            case "integer":
+            case "integer unsigned":

Review Comment:
   `byte unsigned`, `short unsigned`, `integer unsigned`, and `long unsigned` are all matched through independent case branches. The mapping logic is completely consistent with that of their corresponding signed types, yet the branches are written repeatedly. When adding new unsigned types, additional cases need to be added, resulting in extremely low code scalability. We should preprocess the unsigned type to reduce switch branch logic.
   ```
   if (normalizedType.endsWith ("unsigned")) {
        baseType = normalizedType.replace ("unsigned", "").trim ();
   }
   
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][seatunnel-api] Integrate Gravitino as metadata service for non-relational connectors [seatunnel]

Posted by "chl-wxp (via GitHub)" <gi...@apache.org>.
chl-wxp commented on code in PR #10402:
URL: https://github.com/apache/seatunnel/pull/10402#discussion_r2791957106


##########
seatunnel-api/src/main/java/org/apache/seatunnel/api/metalake/TableSchemaDiscoverer.java:
##########
@@ -0,0 +1,239 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.seatunnel.api.metalake;
+
+import org.apache.seatunnel.shade.com.fasterxml.jackson.databind.JsonNode;
+import org.apache.seatunnel.shade.com.google.common.annotations.VisibleForTesting;
+import org.apache.seatunnel.shade.org.apache.commons.lang3.StringUtils;
+
+import org.apache.seatunnel.api.configuration.ReadonlyConfig;
+import org.apache.seatunnel.api.options.ConnectorCommonOptions;
+import org.apache.seatunnel.api.options.EnvCommonOptions;
+import org.apache.seatunnel.api.options.table.CatalogOptions;
+import org.apache.seatunnel.api.options.table.ColumnOptions;
+import org.apache.seatunnel.api.options.table.FieldOptions;
+import org.apache.seatunnel.api.options.table.TableIdentifierOptions;
+import org.apache.seatunnel.api.options.table.TableSchemaOptions;
+import org.apache.seatunnel.api.table.catalog.CatalogTable;
+import org.apache.seatunnel.api.table.catalog.CatalogTableUtil;
+import org.apache.seatunnel.api.table.catalog.TableIdentifier;
+import org.apache.seatunnel.api.table.catalog.TablePath;
+import org.apache.seatunnel.api.table.catalog.TableSchema;
+import org.apache.seatunnel.api.table.factory.TableSourceFactoryContext;
+import org.apache.seatunnel.common.constants.MetaLakeType;
+import org.apache.seatunnel.common.exception.SeaTunnelRuntimeException;
+
+import lombok.extern.slf4j.Slf4j;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.function.Predicate;
+import java.util.stream.Collectors;
+
+import static org.apache.seatunnel.api.table.schema.exception.SchemaEvolutionErrorCode.GET_META_LAKE_TABLE_SCHEMA_FAILED;
+
+@Slf4j
+public class TableSchemaDiscoverer implements AutoCloseable {
+
+    private final ReadonlyConfig envOptions;
+    private final ReadonlyConfig sourceOptions;
+    private final String catalogName;
+    private MetalakeClient metalakeClient;
+    private final MetaLakeTableSchemaConvertor metaLakeTableSchemaConvertor;
+
+    public TableSchemaDiscoverer(TableSourceFactoryContext context, String catalogName) {
+        this.envOptions = context.getEnvOptions();
+        this.sourceOptions = context.getOptions();
+        this.catalogName = catalogName;
+        if (enableMetaLakeClient(context.getOptions())) {
+            this.metalakeClient = MetaLakeFactory.createClient(getMetaLakeType());
+        }
+        this.metaLakeTableSchemaConvertor = MetaLakeFactory.createTypeMapper(getMetaLakeType());
+    }
+
+    @VisibleForTesting
+    protected TableSchemaDiscoverer(
+            ReadonlyConfig envOptions,
+            ReadonlyConfig sourceOptions,
+            String catalogName,
+            MetalakeClient metalakeClient,
+            MetaLakeTableSchemaConvertor convertor) {
+        this.envOptions = envOptions;
+        this.sourceOptions = sourceOptions;
+        this.catalogName = catalogName;
+        this.metalakeClient = metalakeClient;
+        this.metaLakeTableSchemaConvertor = convertor;
+    }
+
+    public List<CatalogTable> discoverTableSchemas() {
+        // schema
+        if (sourceOptions.getOptional(ConnectorCommonOptions.SCHEMA).isPresent()) {
+            return Collections.singletonList(discoverTableSchema(sourceOptions));
+        }
+        // table_config
+        if (sourceOptions.getOptional(TableSchemaOptions.TABLE_CONFIGS).isPresent()) {
+            return sourceOptions.get(TableSchemaOptions.TABLE_CONFIGS).stream()
+                    .map(ReadonlyConfig::fromMap)
+                    .map(this::discoverTableSchema)
+                    .collect(Collectors.toList());
+        }
+        // table_list
+        if (sourceOptions.getOptional(CatalogOptions.TABLE_LIST).isPresent()) {
+            return sourceOptions.get(CatalogOptions.TABLE_LIST).stream()
+                    .map(ReadonlyConfig::fromMap)
+                    .map(this::discoverTableSchema)
+                    .collect(Collectors.toList());
+        }
+        return Collections.singletonList(CatalogTableUtil.buildSimpleTextTable());
+    }
+
+    private CatalogTable discoverTableSchema(ReadonlyConfig sourceOptions) {
+        final Map<String, Object> schemaMap = sourceOptions.get(ConnectorCommonOptions.SCHEMA);
+        ReadonlyConfig schemaConfig = ReadonlyConfig.fromMap(schemaMap);
+        // fields or columns
+        if (schemaConfig.getOptional(ColumnOptions.COLUMNS).isPresent()
+                || sourceOptions.getOptional(FieldOptions.FIELDS).isPresent()) {
+            return discoverTableSchemaFromConfig(sourceOptions);
+        }

Review Comment:
   Check `ColumnOptions.COLUMNS`, `FieldOptions.FIELDS`, `ColumnOptions.SCHEMA_URL` from `sourceOptions.get(ConnectorCommonOptions.SCHEMA)`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][seatunnel-api] Integrate Gravitino as metadata service for non-relational connectors [seatunnel]

Posted by "Carl-Zhou-CN (via GitHub)" <gi...@apache.org>.
Carl-Zhou-CN commented on PR #10402:
URL: https://github.com/apache/seatunnel/pull/10402#issuecomment-3874862271

   good job


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org