You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "Hisoka-X (via GitHub)" <gi...@apache.org> on 2025/06/12 12:09:10 UTC
Re: [PR] [Feature][Jdbc] Support read multiple tables by regular expressions [seatunnel]
Hisoka-X commented on code in PR #9380:
URL: https://github.com/apache/seatunnel/pull/9380#discussion_r2142567414
##########
docs/en/connector-v2/source/Jdbc.md:
##########
@@ -74,6 +74,119 @@ supports query SQL and can achieve projection effect.
| split.string_split_mode | String | No | sample | Supports different string splitting algorithms. By default, `sample` is used to determine the split by sampling the string value. You can switch to `charset_based` to enable charset-based string splitting algorithm. When set to `charset_based`, the algorithm assumes characters of partition_column are within ASCII range 32-126, which covers most character-based splitting scenarios. |
| split.string_split_mode_collate | String | No | - | Specifies the collation to use when string_split_mode is set to `charset_based` and the table has a special collation. If not specified, the database's default collation will be used. |
+### Table Matching
+
+The JDBC Source connector supports two ways to specify tables:
+
+1. **Exact Table Path**: Use `table_path` to specify a single table with its full path.
+ ```hocon
+ table_path = "testdb.table1"
+ ```
+
+2. **Regular Expression**: Use `table_path` with a regex pattern to match multiple tables.
+ ```hocon
+ table_path = "testdb.table\\d+" # Matches table1, table2, table3, etc.
+ use_regex = true
+ ```
+
+#### Regular Expression Support for Table Names
+
+The JDBC connector supports using regular expressions to match multiple tables. This feature allows you to process multiple tables with a single source configuration.
+
+### Configuration
+
+To use regular expression matching for table paths:
+
+1. Set `use_regex = true` to enable regex matching
+2. If `use_regex` is not set or set to `false`, the connector will treat the table_path as an exact path (no regex matching)
+
+### Parameters
+
+| Name | Type | Required | Default | Description |
+|------|------|----------|---------|-------------|
+| use_regex | Boolean | No | false | Control regular expression matching for table_path. When set to `true`, the table_path will be treated as a regular expression pattern. When set to `false` or not specified, the table_path will be treated as an exact path (no regex matching). |
+
+### Regular Expression Syntax Notes
+
+- **Path Separator**: The dot (`.`) is treated as a separator between database, schema, and table names.
+- **Escaped Dots**: If you need to use a dot (`.`) as a wildcard character in your regular expression to match any character, you must escape it with a backslash (`\.`).
+- **Path Format**: For paths like `database.table` or `database.schema.table`, the last unescaped dot separates the table pattern from the database/schema pattern.
+- **Pattern Examples**:
+ - `test.table\\d+` - Matches tables like `table1`, `table2`, etc. in the `test` database
+ - `test\\.table\\d+` - Matches tables with literal dots in names like `test.table1`, `test.table2`
+ - `.*\\.user_.*` - Matches any database with tables starting with `user_`
+
+### Example
+
+```hocon
+source {
+ Jdbc {
+ url = "jdbc:mysql://localhost:3306/test"
+ driver = "com.mysql.cj.jdbc.Driver"
+ user = "root"
+ password = "password"
+
+ table_list = [
+ {
+ # Regex matching - match any table in test database
+ table_path = "test.*"
+ use_regex = true
+ },
+ {
+ # Regex matching - match tables with "user" followed by digits
+ table_path = "test.user\\d+"
+ use_regex = true
+ },
+ {
+ # Exact matching - simple table name
+ table_path = "test.config"
+ # use_regex not specified, defaults to false
+ },
+ ]
+ }
+}
+```
+
+### Notes
+
+- **Path Parsing**: For database paths with multiple parts (e.g., `db.schema.table`), the last unescaped dot separates the table pattern from the database/schema pattern.
+- **Performance**: For performance reasons, a maximum of 1000 tables can be matched by a single regular expression pattern.
+- **Explicit Control**: When `use_regex` is explicitly set to `true` or `false`, it overrides auto-detection. When `use_regex = false`, the table_path is treated as an exact path even if it contains regex-like characters.
+- **Auto-detection**: If `use_regex` is not specified (null), the connector will automatically detect regex patterns by looking for common regex metacharacters like `*`, `+`, `\\d`, `[...]`, `^`, `$`, `()`, `{}`.
+- **Error Handling**: Invalid regex patterns will cause the connector to throw an `IllegalArgumentException` with details about the error.
+- **Case Sensitivity**: Regular expression matching is case-sensitive by default.
Review Comment:
```suggestion
```
##########
docs/en/connector-v2/source/Jdbc.md:
##########
@@ -74,6 +74,119 @@ supports query SQL and can achieve projection effect.
| split.string_split_mode | String | No | sample | Supports different string splitting algorithms. By default, `sample` is used to determine the split by sampling the string value. You can switch to `charset_based` to enable charset-based string splitting algorithm. When set to `charset_based`, the algorithm assumes characters of partition_column are within ASCII range 32-126, which covers most character-based splitting scenarios. |
| split.string_split_mode_collate | String | No | - | Specifies the collation to use when string_split_mode is set to `charset_based` and the table has a special collation. If not specified, the database's default collation will be used. |
+### Table Matching
+
+The JDBC Source connector supports two ways to specify tables:
+
+1. **Exact Table Path**: Use `table_path` to specify a single table with its full path.
+ ```hocon
+ table_path = "testdb.table1"
+ ```
+
+2. **Regular Expression**: Use `table_path` with a regex pattern to match multiple tables.
+ ```hocon
+ table_path = "testdb.table\\d+" # Matches table1, table2, table3, etc.
+ use_regex = true
+ ```
+
+#### Regular Expression Support for Table Names
+
+The JDBC connector supports using regular expressions to match multiple tables. This feature allows you to process multiple tables with a single source configuration.
+
+### Configuration
+
+To use regular expression matching for table paths:
+
+1. Set `use_regex = true` to enable regex matching
+2. If `use_regex` is not set or set to `false`, the connector will treat the table_path as an exact path (no regex matching)
+
+### Parameters
+
+| Name | Type | Required | Default | Description |
+|------|------|----------|---------|-------------|
+| use_regex | Boolean | No | false | Control regular expression matching for table_path. When set to `true`, the table_path will be treated as a regular expression pattern. When set to `false` or not specified, the table_path will be treated as an exact path (no regex matching). |
+
+### Regular Expression Syntax Notes
+
+- **Path Separator**: The dot (`.`) is treated as a separator between database, schema, and table names.
+- **Escaped Dots**: If you need to use a dot (`.`) as a wildcard character in your regular expression to match any character, you must escape it with a backslash (`\.`).
+- **Path Format**: For paths like `database.table` or `database.schema.table`, the last unescaped dot separates the table pattern from the database/schema pattern.
+- **Pattern Examples**:
+ - `test.table\\d+` - Matches tables like `table1`, `table2`, etc. in the `test` database
+ - `test\\.table\\d+` - Matches tables with literal dots in names like `test.table1`, `test.table2`
+ - `.*\\.user_.*` - Matches any database with tables starting with `user_`
+
+### Example
+
+```hocon
+source {
+ Jdbc {
+ url = "jdbc:mysql://localhost:3306/test"
+ driver = "com.mysql.cj.jdbc.Driver"
+ user = "root"
+ password = "password"
+
+ table_list = [
+ {
+ # Regex matching - match any table in test database
+ table_path = "test.*"
+ use_regex = true
+ },
+ {
+ # Regex matching - match tables with "user" followed by digits
+ table_path = "test.user\\d+"
+ use_regex = true
+ },
+ {
+ # Exact matching - simple table name
+ table_path = "test.config"
+ # use_regex not specified, defaults to false
+ },
+ ]
+ }
+}
+```
+
+### Notes
+
+- **Path Parsing**: For database paths with multiple parts (e.g., `db.schema.table`), the last unescaped dot separates the table pattern from the database/schema pattern.
+- **Performance**: For performance reasons, a maximum of 1000 tables can be matched by a single regular expression pattern.
+- **Explicit Control**: When `use_regex` is explicitly set to `true` or `false`, it overrides auto-detection. When `use_regex = false`, the table_path is treated as an exact path even if it contains regex-like characters.
Review Comment:
```suggestion
```
##########
seatunnel-api/src/main/java/org/apache/seatunnel/api/table/catalog/Catalog.java:
##########
@@ -164,15 +164,52 @@ default List<CatalogTable> getTables(ReadonlyConfig config) throws CatalogExcept
Pattern databasePattern =
Pattern.compile(config.get(ConnectorCommonOptions.DATABASE_PATTERN));
Pattern tablePattern = Pattern.compile(config.get(ConnectorCommonOptions.TABLE_PATTERN));
+ Pattern schemaPattern = Pattern.compile(config.get(ConnectorCommonOptions.SCHEMA_PATTERN));
+
+ // Check if schema matching is needed
+ boolean needSchemaMatch = !".*".equals(config.get(ConnectorCommonOptions.SCHEMA_PATTERN));
Review Comment:
Why we need `SCHEMA_PATTERN`?
##########
docs/en/connector-v2/source/Jdbc.md:
##########
@@ -74,6 +74,119 @@ supports query SQL and can achieve projection effect.
| split.string_split_mode | String | No | sample | Supports different string splitting algorithms. By default, `sample` is used to determine the split by sampling the string value. You can switch to `charset_based` to enable charset-based string splitting algorithm. When set to `charset_based`, the algorithm assumes characters of partition_column are within ASCII range 32-126, which covers most character-based splitting scenarios. |
| split.string_split_mode_collate | String | No | - | Specifies the collation to use when string_split_mode is set to `charset_based` and the table has a special collation. If not specified, the database's default collation will be used. |
+### Table Matching
+
+The JDBC Source connector supports two ways to specify tables:
+
+1. **Exact Table Path**: Use `table_path` to specify a single table with its full path.
+ ```hocon
+ table_path = "testdb.table1"
+ ```
+
+2. **Regular Expression**: Use `table_path` with a regex pattern to match multiple tables.
+ ```hocon
+ table_path = "testdb.table\\d+" # Matches table1, table2, table3, etc.
+ use_regex = true
+ ```
+
+#### Regular Expression Support for Table Names
+
+The JDBC connector supports using regular expressions to match multiple tables. This feature allows you to process multiple tables with a single source configuration.
+
+### Configuration
Review Comment:
```suggestion
#### Configuration
```
##########
docs/en/connector-v2/source/Jdbc.md:
##########
@@ -74,6 +74,119 @@ supports query SQL and can achieve projection effect.
| split.string_split_mode | String | No | sample | Supports different string splitting algorithms. By default, `sample` is used to determine the split by sampling the string value. You can switch to `charset_based` to enable charset-based string splitting algorithm. When set to `charset_based`, the algorithm assumes characters of partition_column are within ASCII range 32-126, which covers most character-based splitting scenarios. |
| split.string_split_mode_collate | String | No | - | Specifies the collation to use when string_split_mode is set to `charset_based` and the table has a special collation. If not specified, the database's default collation will be used. |
+### Table Matching
+
+The JDBC Source connector supports two ways to specify tables:
+
+1. **Exact Table Path**: Use `table_path` to specify a single table with its full path.
+ ```hocon
+ table_path = "testdb.table1"
+ ```
+
+2. **Regular Expression**: Use `table_path` with a regex pattern to match multiple tables.
+ ```hocon
+ table_path = "testdb.table\\d+" # Matches table1, table2, table3, etc.
+ use_regex = true
+ ```
+
+#### Regular Expression Support for Table Names
+
+The JDBC connector supports using regular expressions to match multiple tables. This feature allows you to process multiple tables with a single source configuration.
+
+### Configuration
+
+To use regular expression matching for table paths:
+
+1. Set `use_regex = true` to enable regex matching
+2. If `use_regex` is not set or set to `false`, the connector will treat the table_path as an exact path (no regex matching)
+
+### Parameters
+
+| Name | Type | Required | Default | Description |
+|------|------|----------|---------|-------------|
+| use_regex | Boolean | No | false | Control regular expression matching for table_path. When set to `true`, the table_path will be treated as a regular expression pattern. When set to `false` or not specified, the table_path will be treated as an exact path (no regex matching). |
+
+### Regular Expression Syntax Notes
Review Comment:
```suggestion
#### Regular Expression Syntax Notes
```
##########
docs/en/connector-v2/source/Jdbc.md:
##########
@@ -74,6 +74,119 @@ supports query SQL and can achieve projection effect.
| split.string_split_mode | String | No | sample | Supports different string splitting algorithms. By default, `sample` is used to determine the split by sampling the string value. You can switch to `charset_based` to enable charset-based string splitting algorithm. When set to `charset_based`, the algorithm assumes characters of partition_column are within ASCII range 32-126, which covers most character-based splitting scenarios. |
| split.string_split_mode_collate | String | No | - | Specifies the collation to use when string_split_mode is set to `charset_based` and the table has a special collation. If not specified, the database's default collation will be used. |
Review Comment:
Please update optoins list too.
##########
docs/en/connector-v2/source/Jdbc.md:
##########
@@ -74,6 +74,119 @@ supports query SQL and can achieve projection effect.
| split.string_split_mode | String | No | sample | Supports different string splitting algorithms. By default, `sample` is used to determine the split by sampling the string value. You can switch to `charset_based` to enable charset-based string splitting algorithm. When set to `charset_based`, the algorithm assumes characters of partition_column are within ASCII range 32-126, which covers most character-based splitting scenarios. |
| split.string_split_mode_collate | String | No | - | Specifies the collation to use when string_split_mode is set to `charset_based` and the table has a special collation. If not specified, the database's default collation will be used. |
+### Table Matching
+
+The JDBC Source connector supports two ways to specify tables:
+
+1. **Exact Table Path**: Use `table_path` to specify a single table with its full path.
+ ```hocon
+ table_path = "testdb.table1"
+ ```
+
+2. **Regular Expression**: Use `table_path` with a regex pattern to match multiple tables.
+ ```hocon
+ table_path = "testdb.table\\d+" # Matches table1, table2, table3, etc.
+ use_regex = true
+ ```
+
+#### Regular Expression Support for Table Names
+
+The JDBC connector supports using regular expressions to match multiple tables. This feature allows you to process multiple tables with a single source configuration.
+
+### Configuration
+
+To use regular expression matching for table paths:
+
+1. Set `use_regex = true` to enable regex matching
+2. If `use_regex` is not set or set to `false`, the connector will treat the table_path as an exact path (no regex matching)
+
+### Parameters
+
+| Name | Type | Required | Default | Description |
+|------|------|----------|---------|-------------|
+| use_regex | Boolean | No | false | Control regular expression matching for table_path. When set to `true`, the table_path will be treated as a regular expression pattern. When set to `false` or not specified, the table_path will be treated as an exact path (no regex matching). |
+
+### Regular Expression Syntax Notes
+
+- **Path Separator**: The dot (`.`) is treated as a separator between database, schema, and table names.
+- **Escaped Dots**: If you need to use a dot (`.`) as a wildcard character in your regular expression to match any character, you must escape it with a backslash (`\.`).
+- **Path Format**: For paths like `database.table` or `database.schema.table`, the last unescaped dot separates the table pattern from the database/schema pattern.
+- **Pattern Examples**:
+ - `test.table\\d+` - Matches tables like `table1`, `table2`, etc. in the `test` database
+ - `test\\.table\\d+` - Matches tables with literal dots in names like `test.table1`, `test.table2`
+ - `.*\\.user_.*` - Matches any database with tables starting with `user_`
+
+### Example
+
+```hocon
+source {
+ Jdbc {
+ url = "jdbc:mysql://localhost:3306/test"
+ driver = "com.mysql.cj.jdbc.Driver"
+ user = "root"
+ password = "password"
+
+ table_list = [
+ {
+ # Regex matching - match any table in test database
+ table_path = "test.*"
+ use_regex = true
+ },
+ {
+ # Regex matching - match tables with "user" followed by digits
+ table_path = "test.user\\d+"
+ use_regex = true
+ },
+ {
+ # Exact matching - simple table name
+ table_path = "test.config"
+ # use_regex not specified, defaults to false
+ },
+ ]
+ }
+}
+```
+
+### Notes
+
+- **Path Parsing**: For database paths with multiple parts (e.g., `db.schema.table`), the last unescaped dot separates the table pattern from the database/schema pattern.
+- **Performance**: For performance reasons, a maximum of 1000 tables can be matched by a single regular expression pattern.
Review Comment:
why we need this limit?
##########
docs/en/connector-v2/source/Jdbc.md:
##########
@@ -74,6 +74,119 @@ supports query SQL and can achieve projection effect.
| split.string_split_mode | String | No | sample | Supports different string splitting algorithms. By default, `sample` is used to determine the split by sampling the string value. You can switch to `charset_based` to enable charset-based string splitting algorithm. When set to `charset_based`, the algorithm assumes characters of partition_column are within ASCII range 32-126, which covers most character-based splitting scenarios. |
| split.string_split_mode_collate | String | No | - | Specifies the collation to use when string_split_mode is set to `charset_based` and the table has a special collation. If not specified, the database's default collation will be used. |
+### Table Matching
+
+The JDBC Source connector supports two ways to specify tables:
+
+1. **Exact Table Path**: Use `table_path` to specify a single table with its full path.
+ ```hocon
+ table_path = "testdb.table1"
+ ```
+
+2. **Regular Expression**: Use `table_path` with a regex pattern to match multiple tables.
+ ```hocon
+ table_path = "testdb.table\\d+" # Matches table1, table2, table3, etc.
+ use_regex = true
+ ```
+
+#### Regular Expression Support for Table Names
+
+The JDBC connector supports using regular expressions to match multiple tables. This feature allows you to process multiple tables with a single source configuration.
+
+### Configuration
+
+To use regular expression matching for table paths:
+
+1. Set `use_regex = true` to enable regex matching
+2. If `use_regex` is not set or set to `false`, the connector will treat the table_path as an exact path (no regex matching)
+
+### Parameters
Review Comment:
```suggestion
#### Parameters
```
##########
docs/en/connector-v2/source/Jdbc.md:
##########
@@ -74,6 +74,119 @@ supports query SQL and can achieve projection effect.
| split.string_split_mode | String | No | sample | Supports different string splitting algorithms. By default, `sample` is used to determine the split by sampling the string value. You can switch to `charset_based` to enable charset-based string splitting algorithm. When set to `charset_based`, the algorithm assumes characters of partition_column are within ASCII range 32-126, which covers most character-based splitting scenarios. |
| split.string_split_mode_collate | String | No | - | Specifies the collation to use when string_split_mode is set to `charset_based` and the table has a special collation. If not specified, the database's default collation will be used. |
+### Table Matching
+
+The JDBC Source connector supports two ways to specify tables:
+
+1. **Exact Table Path**: Use `table_path` to specify a single table with its full path.
+ ```hocon
+ table_path = "testdb.table1"
+ ```
+
+2. **Regular Expression**: Use `table_path` with a regex pattern to match multiple tables.
+ ```hocon
+ table_path = "testdb.table\\d+" # Matches table1, table2, table3, etc.
+ use_regex = true
+ ```
+
+#### Regular Expression Support for Table Names
+
+The JDBC connector supports using regular expressions to match multiple tables. This feature allows you to process multiple tables with a single source configuration.
+
+### Configuration
+
+To use regular expression matching for table paths:
+
+1. Set `use_regex = true` to enable regex matching
+2. If `use_regex` is not set or set to `false`, the connector will treat the table_path as an exact path (no regex matching)
+
+### Parameters
+
+| Name | Type | Required | Default | Description |
+|------|------|----------|---------|-------------|
+| use_regex | Boolean | No | false | Control regular expression matching for table_path. When set to `true`, the table_path will be treated as a regular expression pattern. When set to `false` or not specified, the table_path will be treated as an exact path (no regex matching). |
+
+### Regular Expression Syntax Notes
+
+- **Path Separator**: The dot (`.`) is treated as a separator between database, schema, and table names.
+- **Escaped Dots**: If you need to use a dot (`.`) as a wildcard character in your regular expression to match any character, you must escape it with a backslash (`\.`).
+- **Path Format**: For paths like `database.table` or `database.schema.table`, the last unescaped dot separates the table pattern from the database/schema pattern.
+- **Pattern Examples**:
+ - `test.table\\d+` - Matches tables like `table1`, `table2`, etc. in the `test` database
+ - `test\\.table\\d+` - Matches tables with literal dots in names like `test.table1`, `test.table2`
+ - `.*\\.user_.*` - Matches any database with tables starting with `user_`
+
+### Example
+
+```hocon
+source {
+ Jdbc {
+ url = "jdbc:mysql://localhost:3306/test"
+ driver = "com.mysql.cj.jdbc.Driver"
+ user = "root"
+ password = "password"
+
+ table_list = [
+ {
+ # Regex matching - match any table in test database
+ table_path = "test.*"
+ use_regex = true
+ },
+ {
+ # Regex matching - match tables with "user" followed by digits
+ table_path = "test.user\\d+"
+ use_regex = true
+ },
+ {
+ # Exact matching - simple table name
+ table_path = "test.config"
+ # use_regex not specified, defaults to false
+ },
+ ]
+ }
+}
+```
+
+### Notes
+
+- **Path Parsing**: For database paths with multiple parts (e.g., `db.schema.table`), the last unescaped dot separates the table pattern from the database/schema pattern.
+- **Performance**: For performance reasons, a maximum of 1000 tables can be matched by a single regular expression pattern.
+- **Explicit Control**: When `use_regex` is explicitly set to `true` or `false`, it overrides auto-detection. When `use_regex = false`, the table_path is treated as an exact path even if it contains regex-like characters.
+- **Auto-detection**: If `use_regex` is not specified (null), the connector will automatically detect regex patterns by looking for common regex metacharacters like `*`, `+`, `\\d`, `[...]`, `^`, `$`, `()`, `{}`.
Review Comment:
I think we can not keep this feature. Let user set `user_regex` if them want parse table path by regex.
##########
docs/en/connector-v2/source/Jdbc.md:
##########
@@ -74,6 +74,119 @@ supports query SQL and can achieve projection effect.
| split.string_split_mode | String | No | sample | Supports different string splitting algorithms. By default, `sample` is used to determine the split by sampling the string value. You can switch to `charset_based` to enable charset-based string splitting algorithm. When set to `charset_based`, the algorithm assumes characters of partition_column are within ASCII range 32-126, which covers most character-based splitting scenarios. |
| split.string_split_mode_collate | String | No | - | Specifies the collation to use when string_split_mode is set to `charset_based` and the table has a special collation. If not specified, the database's default collation will be used. |
+### Table Matching
+
+The JDBC Source connector supports two ways to specify tables:
+
+1. **Exact Table Path**: Use `table_path` to specify a single table with its full path.
+ ```hocon
+ table_path = "testdb.table1"
+ ```
+
+2. **Regular Expression**: Use `table_path` with a regex pattern to match multiple tables.
+ ```hocon
+ table_path = "testdb.table\\d+" # Matches table1, table2, table3, etc.
+ use_regex = true
+ ```
+
+#### Regular Expression Support for Table Names
+
+The JDBC connector supports using regular expressions to match multiple tables. This feature allows you to process multiple tables with a single source configuration.
+
+### Configuration
+
+To use regular expression matching for table paths:
+
+1. Set `use_regex = true` to enable regex matching
+2. If `use_regex` is not set or set to `false`, the connector will treat the table_path as an exact path (no regex matching)
+
+### Parameters
+
+| Name | Type | Required | Default | Description |
+|------|------|----------|---------|-------------|
+| use_regex | Boolean | No | false | Control regular expression matching for table_path. When set to `true`, the table_path will be treated as a regular expression pattern. When set to `false` or not specified, the table_path will be treated as an exact path (no regex matching). |
+
+### Regular Expression Syntax Notes
+
+- **Path Separator**: The dot (`.`) is treated as a separator between database, schema, and table names.
+- **Escaped Dots**: If you need to use a dot (`.`) as a wildcard character in your regular expression to match any character, you must escape it with a backslash (`\.`).
+- **Path Format**: For paths like `database.table` or `database.schema.table`, the last unescaped dot separates the table pattern from the database/schema pattern.
+- **Pattern Examples**:
+ - `test.table\\d+` - Matches tables like `table1`, `table2`, etc. in the `test` database
+ - `test\\.table\\d+` - Matches tables with literal dots in names like `test.table1`, `test.table2`
+ - `.*\\.user_.*` - Matches any database with tables starting with `user_`
+
+### Example
Review Comment:
```suggestion
#### Example
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org