You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "chl-wxp (via GitHub)" <gi...@apache.org> on 2026/01/14 07:38:17 UTC
[I] [Feature][seatunnel-api] Non-relational databases use Gravitino's metadata restApi instead of schema configuration [seatunnel]
chl-wxp opened a new issue, #10339:
URL: https://github.com/apache/seatunnel/issues/10339
### Search before asking
- [x] I had searched in the [feature](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22Feature%22) and found no similar feature requirement.
### Description
1
### Usage Scenario
_No response_
### Related issues
_No response_
### Are you willing to submit a PR?
- [x] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature][seatunnel-api] Non-relational databases use Gravitino's metadata restApi instead of schema configuration [seatunnel]
Posted by "chl-wxp (via GitHub)" <gi...@apache.org>.
chl-wxp commented on issue #10339:
URL: https://github.com/apache/seatunnel/issues/10339#issuecomment-3752933792
## Core Idea
For **semi-structured and unstructured data sources** (such as FTP, S3, file systems, etc.),
SeaTunnel can resolve table schemas from **Gravitino REST APIs**.
This does **not** replace the existing inline `schema` configuration.
It is a **new optional mechanism** that is **fully backward compatible**.
The priority order is:
### 1 If `schema` Is Defined, Always Use It
If a connector configuration contains a `schema` block, SeaTunnel **must ignore Gravitino**
and use the inline schema directly.
```hocon
FtpFile {
path = "/tmp/seatunnel/sink/text"
host = "192.168.31.48"
port = 21
user = tyrantlucifer
password = tianchao
file_format_type = "text"
schema = {
name = string
age = int
}
field_delimiter = "#"
}
```
### 2 Using Gravitino via env (Recommended Mode)
SeaTunnel already integrates with Gravitino Metalake at the engine level.
When configured in env, all non-relational sources can reference schemas by name.
```hocon
env {
metalake_enabled = true
metalake_type = "gravitino"
metalake_url = "http://localhost:8090/api/metalakes/metalake_name/catalogs/"
}
```
#### 2.1Use schema_path
```hocon
FtpFile {
path = "/tmp/seatunnel/sink/text"
host = "192.168.31.48"
port = 21
user = tyrantlucifer
password = tianchao
file_format_type = "text"
schema_path = "catalog_name.ykw.test_table"
field_delimiter = "#"
}
```
#### 2.2 Use schema_url
```hocon
FtpFile {
path = "/tmp/seatunnel/sink/text"
host = "192.168.31.48"
port = 21
user = tyrantlucifer
password = tianchao
file_format_type = "text"
schema_url = "http://localhost:8090/api/metalakes/laowang_test/catalogs/221-pgsql/schemas/ykw/tables/all_type"
field_delimiter = "#"
}
```
### 3 Fallback to OS Environment Variables
If Gravitino is not defined in env, SeaTunnel reads from OS environment variables:
```hocon
metalake_enabled
metalake_type
metalake_url
```
The behavior is identical to the env configuration in Section 2.
### 4 Standalone Gravitino Configuration at Connector Level
If no metadata center is configured globally, the connector can define Gravitino directly.
#### 4.1 Using schema_url
```hocon
FtpFile {
path = "/tmp/seatunnel/sink/text"
host = "192.168.31.48"
port = 21
user = tyrantlucifer
password = tianchao
file_format_type = "text"
metalake_type = "gravitino"
schema_url = "http://localhost:8090/api/metalakes/laowang_test/catalogs/221-pgsql/schemas/ykw/tables/all_type"
field_delimiter = "#"
}
```
#### 4.2 Using schema_path
```hocon
FtpFile {
path = "/tmp/seatunnel/sink/text"
host = "192.168.31.48"
port = 21
user = tyrantlucifer
password = tianchao
file_format_type = "text"
metalake_type = "gravitino"
metalake_url = "http://localhost:8090/api/metalakes/metalake_name/catalogs/"
schema_path = "catalog_name.ykw.test_table"
field_delimiter = "#"
}
```
### 5 Find the http detector of restApi according to metalake_type
### 6 The detector calls the spliced URL to get the responseBody, hands it to the mapper for type matching, and completes the catalogTable construction.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature][seatunnel-api] Non-relational databases use Gravitino's metadata restApi instead of schema configuration [seatunnel]
Posted by "davidzollo (via GitHub)" <gi...@apache.org>.
davidzollo commented on issue #10339:
URL: https://github.com/apache/seatunnel/issues/10339#issuecomment-3749282451
Good suggestion.
Can you submit a design for this feature? like this issue https://github.com/apache/seatunnel/issues/10196
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [STIP-22] [Feature][seatunnel-api] Non-relational databases use Gravitino's metadata restApi instead of schema configuration [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on issue #10339:
URL: https://github.com/apache/seatunnel/issues/10339#issuecomment-4236689721
Following up here as Daniel:
this feature has landed on `dev` via #10402, so closing this issue was reasonable. The merged change adds the Gravitino-backed metadata path for non-relational connectors and keeps the existing inline schema path as the backward-compatible fallback.
If anyone continues from this direction, the best next step is now a focused follow-up issue for any missing connector coverage or docs gaps, rather than reopening this design thread.
Thanks for driving the proposal and implementation forward.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature][seatunnel-api] Non-relational databases use Gravitino's metadata restApi instead of schema configuration [seatunnel]
Posted by "chl-wxp (via GitHub)" <gi...@apache.org>.
chl-wxp commented on issue #10339:
URL: https://github.com/apache/seatunnel/issues/10339#issuecomment-3752889356
```mermaid
flowchart TD
%% ========= Entry =========
A[Start: Non-Relational Source Initialization] --> B{Is 'schema' defined in config?}
%% ========= Step 1: Local Schema =========
B -- Yes --> S1[Use embedded schema from config]
S1 --> Z[Build CatalogTable and finish]
%% ========= Step 2: No local schema =========
B -- No --> C{Is Metalake configured in env?}
%% ========= Step 2a: Seatunnel env =========
C -- Yes --> E1[Read metalake_enabled / metalake_type / metalake_url from env]
%% ========= Step 3: OS env =========
C -- No --> D{Are Metalake variables in OS env?}
D -- Yes --> E2[Read metalake_enabled / metalake_type / metalake_url from OS env]
%% ========= Step 4: Connector config =========
D -- No --> E3[Read metalake_type / metalake_url from connector config]
%% ========= Unified metadata path =========
E1 --> F
E2 --> F
E3 --> F
%% ========= Schema reference =========
F{Is schema_path or schema_url defined?}
F -- schema_url --> G1[Use schema_url directly]
F -- schema_path --> G2[Compose full schema_url using metalake_url + schema_path]
%% ========= Step 5 =========
G1 --> H
G2 --> H
H[Select REST detector by metalake_type]
%% ========= Step 6 =========
H --> I[Call REST API and get responseBody]
I --> J[Type Mapper converts response to internal types]
J --> K[Build CatalogTable from mapped schema]
%% ========= End =========
K --> Z[Schema ready for Source execution]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature][seatunnel-api] Non-relational databases use Gravitino's metadata restApi instead of schema configuration [seatunnel]
Posted by "chl-wxp (via GitHub)" <gi...@apache.org>.
chl-wxp commented on issue #10339:
URL: https://github.com/apache/seatunnel/issues/10339#issuecomment-3752705420
```mermaid
flowchart TD
A["Parse job config"] --> B{"Has schema_url or schema_path"}
B -- "No" --> C["Use legacy schema config or connector defaults"]
C --> Z["Submit job"]
B -- "Yes" --> D["Resolve metalake config"]
D --> D1["From env metalake settings"]
D1 --> D2["Else from OS env metalake settings"]
D2 --> D3["Else from connector metalake settings"]
D3 --> E{"Has schema_url"}
E -- "Yes" --> F["Final url equals schema_url"]
E -- "No" --> G["Parse schema_path into three parts"]
G --> H{"Has metalake_url"}
H -- "No" --> H1["Fail missing metalake_url for schema_path"]
H -- "Yes" --> I["Build final url from base url and schema_path"]
F --> J["Select detector by metalake_type"]
I --> J
J --> K{"Detector exists"}
K -- "No" --> K1["Fail unknown metalake_type"]
K -- "Yes" --> L["HTTP fetch schema from final url"]
L --> M{"HTTP success"}
M -- "No" --> M1["Fail http error or timeout"]
M -- "Yes" --> N["Map response to SeaTunnel types"]
N --> O{"Mapping success"}
O -- "No" --> O1["Fail parse or type mapping error"]
O -- "Yes" --> P["Build CatalogTable"]
P --> Z["Submit job"]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature][seatunnel-api] Non-relational databases use Gravitino's metadata restApi instead of schema configuration [seatunnel]
Posted by "davidzollo (via GitHub)" <gi...@apache.org>.
davidzollo closed issue #10339: [Feature][seatunnel-api] Non-relational databases use Gravitino's metadata restApi instead of schema configuration
URL: https://github.com/apache/seatunnel/issues/10339
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature][seatunnel-api] Non-relational databases use Gravitino's metadata restApi instead of schema configuration [seatunnel]
Posted by "chl-wxp (via GitHub)" <gi...@apache.org>.
chl-wxp commented on issue #10339:
URL: https://github.com/apache/seatunnel/issues/10339#issuecomment-3755306048
> Thanks for the proposal. This is a valid feature gap: current Metalake integration only supports placeholder replacement for credentials, not schema retrieval from Gravitino.
>
> **Evidence:**
>
> * `seatunnel-api/src/main/java/org/apache/seatunnel/api/metalake/MetalakeConfigUtils.java` (L76-84) only fetches properties for placeholder substitution
> * `seatunnel-connectors-v2/connector-mongodb/src/main/java/org/apache/seatunnel/connectors/seatunnel/mongodb/source/MongodbSourceFactory.java` (L47-52) requires inline `schema` as mandatory
> * `seatunnel-connectors-v2/connector-file/connector-file-base/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/config/BaseFileSourceConfig.java` (L83-89) uses inline schema or falls back to simple text table
>
> **Suggested starting points:**
>
> 1. `seatunnel-api/src/main/java/org/apache/seatunnel/api/options/table/TableSchemaOptions.java` — Add `schema_path`, `schema_url` options
> 2. `seatunnel-api/src/main/java/org/apache/seatunnel/api/metalake/MetalakeClient.java` — Add `getTableSchema(String tableRef)` method
> 3. `seatunnel-api/src/main/java/org/apache/seatunnel/api/metalake/GravitinoClient.java` — Implement REST call to `/tables/{table}`
> 4. `seatunnel-connectors-v2/connector-file/connector-file-base/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/config/BaseFileSourceConfig.java` — Extend schema resolution logic
>
> **Questions to clarify:**
>
> 1. Should non-relational catalogs (File/ES/MongoDB) use new Gravitino catalog providers or reuse existing `fileset`/`messaging` types?
> 2. How should complex types (ES nested/object, MongoDB documents) be mapped from Gravitino to SeaTunnel data types?
1. To implement schema detection, try to rely on the existing GravitinoClient.
2. Gravitino is a new function option, and the schema will not be replaced.
3. Gravitino is not only a data source center, but also plays a more important role as a metadata center, which unifies data types and can serve as the only source of truth for data types in non-relational databases. I mentioned in the design document that after requesting the restApi of the schema, I will not process it based on the placeholder. The correct approach is to map the Gravitino type and the seatunnel type. The following is an example of the request result when I obtain a pgsql table structure:
```json
{
"name": "id",
"type": "integer",
"nullable": true,
"autoIncrement": false,
"defaultValue": {
"type": "literal",
"dataType": "null",
"value": "NULL"
}
},
{
"name": "big_number",
"type": "long",
"nullable": true,
"autoIncrement": false,
"defaultValue": {
"type": "literal",
"dataType": "null",
"value": "NULL"
}
},
{
"name": "small_number",
"type": "integer",
"nullable": true,
"autoIncrement": false,
"defaultValue": {
"type": "literal",
"dataType": "null",
"value": "NULL"
}
},
{
"name": "tiny_number",
"type": "short",
"nullable": true,
"autoIncrement": false,
"defaultValue": {
"type": "literal",
"dataType": "null",
"value": "NULL"
}
}
```
It can be seen that Gravitino actually unifies the types, which are not the types of the original database, and I have asked the official for confirmation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature][seatunnel-api] Non-relational databases use Gravitino's metadata restApi instead of schema configuration [seatunnel]
Posted by "zhangshenghang (via GitHub)" <gi...@apache.org>.
zhangshenghang commented on issue #10339:
URL: https://github.com/apache/seatunnel/issues/10339#issuecomment-3755081044
<!-- code-pr-reviewer -->
Thanks for the proposal. This is a valid feature gap: current Metalake integration only supports placeholder replacement for credentials, not schema retrieval from Gravitino.
**Evidence:**
- `seatunnel-api/src/main/java/org/apache/seatunnel/api/metalake/MetalakeConfigUtils.java` (L76-84) only fetches properties for placeholder substitution
- `seatunnel-connectors-v2/connector-mongodb/src/main/java/org/apache/seatunnel/connectors/seatunnel/mongodb/source/MongodbSourceFactory.java` (L47-52) requires inline `schema` as mandatory
- `seatunnel-connectors-v2/connector-file/connector-file-base/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/config/BaseFileSourceConfig.java` (L83-89) uses inline schema or falls back to simple text table
**Suggested starting points:**
1. `seatunnel-api/src/main/java/org/apache/seatunnel/api/options/table/TableSchemaOptions.java` — Add `schema_path`, `schema_url` options
2. `seatunnel-api/src/main/java/org/apache/seatunnel/api/metalake/MetalakeClient.java` — Add `getTableSchema(String tableRef)` method
3. `seatunnel-api/src/main/java/org/apache/seatunnel/api/metalake/GravitinoClient.java` — Implement REST call to `/tables/{table}`
4. `seatunnel-connectors-v2/connector-file/connector-file-base/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/config/BaseFileSourceConfig.java` — Extend schema resolution logic
**Questions to clarify:**
1. Should non-relational catalogs (File/ES/MongoDB) use new Gravitino catalog providers or reuse existing `fileset`/`messaging` types?
2. How should complex types (ES nested/object, MongoDB documents) be mapped from Gravitino to SeaTunnel data types?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature][seatunnel-api] Non-relational databases use Gravitino's metadata restApi instead of schema configuration [seatunnel]
Posted by "chl-wxp (via GitHub)" <gi...@apache.org>.
chl-wxp commented on issue #10339:
URL: https://github.com/apache/seatunnel/issues/10339#issuecomment-3752662746
flowchart TD
A[Start: Parse Job Config] --> B{Source has schema_url or schema_path?}
B -- No --> C[Use legacy local schema config<br/>schema { fields { ... } } / connector default]
C --> Z[Submit Job]
B -- Yes --> D[Resolve Metalake Config]
D --> D1[Priority: env metalake_*]
D1 --> D2[Else: OS env metalake_*]
D2 --> D3[Else: connector metalake_type/metalake_url]
D3 --> E{schema_url provided?}
E -- Yes --> F[FinalSchemaUrl = schema_url]
E -- No --> G[Parse schema_path: catalog.schema.table]
G --> H{metalake_url resolvable?}
H -- No --> H1[Fail: missing metalake_url for schema_path]
H -- Yes --> I[Build FinalSchemaUrl = metalake_url + catalog + /schemas/ + schema + /tables/ + table]
F --> J[Select detector by metalake_type]
I --> J
J --> K{Detector exists?}
K -- No --> K1[Fail: unknown metalake_type]
K -- Yes --> L[HTTP GET FinalSchemaUrl]
L --> M{HTTP OK?}
M -- No --> M1[Fail: HTTP error / timeout]
M -- Yes --> N[Parse response body]
N --> O{Mapping success?}
O -- No --> O1[Fail: parse/type mapping error]
O -- Yes --> P[Build CatalogTable / SeaTunnel schema]
P --> Q[Inject resolved schema into source runtime config]
Q --> Z[Submit Job]
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature][seatunnel-api] Non-relational databases use Gravitino's metadata restApi instead of schema configuration [seatunnel]
Posted by "chl-wxp (via GitHub)" <gi...@apache.org>.
chl-wxp commented on issue #10339:
URL: https://github.com/apache/seatunnel/issues/10339#issuecomment-3752666181
```mermaid
flowchart TD
A[Start: Parse Job Config] --> B{Source has schema_url or schema_path?}
B -- No --> C[Use legacy local schema config<br/>schema { fields { ... } } / connector default]
C --> Z[Submit Job]
B -- Yes --> D[Resolve Metalake Config]
D --> D1[Priority: env metalake_*]
D1 --> D2[Else: OS env metalake_*]
D2 --> D3[Else: connector metalake_type/metalake_url]
D3 --> E{schema_url provided?}
E -- Yes --> F[FinalSchemaUrl = schema_url]
E -- No --> G[Parse schema_path: catalog.schema.table]
G --> H{metalake_url resolvable?}
H -- No --> H1[Fail: missing metalake_url for schema_path]
H -- Yes --> I[Build FinalSchemaUrl = metalake_url + catalog + /schemas/ + schema + /tables/ + table]
F --> J[Select detector by metalake_type]
I --> J
J --> K{Detector exists?}
K -- No --> K1[Fail: unknown metalake_type]
K -- Yes --> L[HTTP GET FinalSchemaUrl]
L --> M{HTTP OK?}
M -- No --> M1[Fail: HTTP error / timeout]
M -- Yes --> N[Parse response body]
N --> O{Mapping success?}
O -- No --> O1[Fail: parse/type mapping error]
O -- Yes --> P[Build CatalogTable / SeaTunnel schema]
P --> Q[Inject resolved schema into source runtime config]
Q --> Z[Submit Job]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org