[I] Add bidirectional Apache Gravitino converter [ossie]

["LiJie20190102 (via GitHub)" <gi...@apache.org>]

LiJie20190102 opened a new issue, #256:
URL: https://github.com/apache/ossie/issues/256

   Hi all 👋
   
   I'd like to gauge interest in adding a **bidirectional converter** between Apache Ossie semantic models and [Apache Gravitino](https://gravitino.apache.org/) under `converters/gravitino/`.
   
   ## Background
   
   [Apache Gravitino](https://gravitino.apache.org/) is a high-performance, geo-distributed, federated metadata lake (Apache Incubating). It provides a unified REST API to manage metadata across many data sources — Hive, JDBC (MySQL/PostgreSQL/Doris/StarRocks), Iceberg, Paimon, Hudi, Kafka, and file-based sources — through a single "catalog of catalogs" abstraction.
   
   Gravitino already knows about **physical metadata** (catalogs, schemas, tables, columns, types, partition keys, tags, properties) for the entire big-data ecosystem. What it does **not** carry is Ossie's **semantic layer** — metrics, cross-dataset relationships, time-dimension markers, and AI context. A converter between the two would let any Ossie-compatible BI/AI tool access the full inventory of a big-data platform through Gravitino's unified API, and conversely let a semantic model authored in Ossie be materialized as table metadata in a big-data catalog.
   
   This is also a **two-Apache-podlings integration** — both Gravitino and Ossie are Apache Incubating projects, and a converter between them benefits both ecosystems.
   
   **I'd like to contribute this converter myself.** I'm a big-data platform engineer with deep experience in the Hive/Spark/Iceberg ecosystem, and I've already studied the existing `converters/polaris/` codebase and prototyped the Gravitino REST API integration. I'm happy to follow the "Contributing a New Converter" steps (vendor entry, both directions, tests, README) and would open a PR once the approach is agreed.
   
   ## What's included
   
   - **`converters/gravitino/`** — a Java/Maven module following the pattern of `converters/polaris/`.
   - `GravitinoClient` — a thin REST client calling Gravitino's existing API (Java 11 `HttpClient` + Jackson), shared by both directions:
     - `GET /api/metalakes/{m}/catalogs` → list catalogs
     - `GET /api/metalakes/{m}/catalogs/{c}/schemas` → list schemas
     - `GET /api/metalakes/{m}/catalogs/{c}/schemas/{s}/tables` → list tables
     - `GET /api/metalakes/{m}/catalogs/{c}/schemas/{s}/tables/{t}` → load table
     - `POST /api/metalakes/{m}/catalogs/{c}/schemas/{s}/tables` → create table
   
   ### Import (Gravitino → Ossie)
   
   Reads table/schema metadata from a Gravitino metalake and produces an Ossie semantic model YAML.
   
   | Gravitino concept | Ossie concept |
   |---|---|
   | Metalake + Catalog | `SemanticModel` |
   | Schema | dataset grouping within the model |
   | Table | `Dataset` |
   | `table.name` | `dataset.name` |
   | `catalog.schema.table` | `dataset.source` |
   | `table.columns[]` | `dataset.fields[]` |
   | `column.name` + `column.type` | `field.name` + `expression` (ANSI_SQL) |
   | `column.type` = timestamp/date | `field.dimension.is_time = true` |
   | `table.partitionKeys[]` | flagged in `field.description` or `custom_extensions` |
   | `table.properties` | `dataset.custom_extensions[vendor_name: GRAVITINO]` |
   | `table.tags` | candidate source for `dataset.ai_context` |
   
   CLI: `ossie-gravitino-converter import --url URL --metalake M [--catalog C] [--auth ...] -o output.yaml`
   
   ### Export (Ossie → Gravitino)
   
   Takes an Ossie semantic model and creates/updates the corresponding tables in Gravitino via its REST API.
   
   | Ossie concept | Gravitino concept |
   |---|---|
   | `SemanticModel` | Metalake + Catalog (1:1, or configurable mapping) |
   | `Dataset` | `Table` |
   | `dataset.name` | `table.name` |
   | `dataset.source` | resolved to `catalog.schema.table` |
   | `dataset.fields[]` | `table.columns[]` |
   | `field.expression` (ANSI_SQL) | `column.name` (extracted from expression) |
   | `field.dimension.is_time` | `column.type` set to `timestamp` / `date` |
   | `dataset.primary_key` | `table.primaryKey` or `table.constraints` |
   | `dataset.unique_keys` | `table.constraints` (unique) |
   | `dataset.custom_extensions[vendor_name: GRAVITINO]` | `table.properties` (round-trip) |
   | `dataset.description` / `field.description` | `table.comment` / `column.comment` |
   
   CLI: `ossie-gravitino-converter export --url URL --metalake M --catalog C ossie_model.yaml [--auth ...] [--dry-run]`
   
   ## Design considerations
   
   - **Round-trip fidelity**: `Gravitino → Ossie → Gravitino` should be lossless. Gravitino's relational model does not have a native "relationship" or "metric" concept, so Ossie relationships and metrics are serialized into `custom_extensions[vendor_name: OSSIE]` on the corresponding table, following the round-trip-fidelity pattern used by the Databricks converter (#224).
   - **No changes on the Gravitino side**: the converter is a pure client of Gravitino's existing REST API. Gravitino does not need to know about Ossie.
   - **Idempotency**: the exporter checks if a table already exists before creating (GET then POST/PUT), so re-running is safe.
   - **Type inference**: Ossie fields are expression-based and don't carry explicit physical types. The exporter uses heuristics (field name patterns + `is_time` flag) to infer Gravitino column types, similar to `PolarisExporter.inferIcebergType()`. Gravitino's `External Type` fallback is used when inference fails.
   - **Big-data ecosystem gap**: Ossie currently has converters for BI/SaaS tools (dbt, Snowflake, Tableau, GoodData, Omni, …) but **none for big-data catalog/metadata services**. Gravitino is the natural bridge — one converter unlocks Hive, Iceberg, Paimon, Hudi, JDBC, and more.
   
   ## Approach
   
   I'd follow the existing converter pattern (closest analogue: `converters/polaris/`):
   
   1. `GravitinoClient` — REST client authenticating via Gravitino's OAuth or basic auth.
   2. `GravitinoImporter` — walks metalake → catalog → schema → table, maps each table to an Ossie `Dataset` with fields derived from columns. Time types (`timestamp`, `date`, `timestamptz`) set `dimension.is_time`.
   3. `GravitinoExporter` — reads an Ossie model, infers column types, and calls Gravitino's create-table API.
   4. Reuse `OsiYamlGenerator` / `OsiModelParser` (or the shared `ossie-common` module if #235 lands first).
   5. Gravitino tags/properties that carry semantic hints (e.g. `description`, `owner`) flow into `ai_context` or `description`; non-semantic properties go into `custom_extensions[vendor_name: GRAVITINO]`.
   6. CLI + tests (mocked Gravitino API responses with Hive + JDBC-MySQL fixtures) + README, following CONTRIBUTING.md.
   
   Before I open the PR: **is there anything you'd want me to keep in mind on approach or scope?** And should I wait for #235 (Java shared module) to land, or build against the current `converters/polaris/` structure for now?
   
   I'll go ahead and start the implementation either way — just want to make sure I'm aligned with the project's direction.
   
   Thanks!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ossie.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Add bidirectional Apache Gravitino converter [ossie]

["wForget (via GitHub)" <gi...@apache.org>]

wForget commented on issue #256:
URL: https://github.com/apache/ossie/issues/256#issuecomment-5065505487

   > Before I open the PR: is there anything you'd want me to keep in mind on approach or scope? And should I wait for https://github.com/apache/ossie/pull/235 (Java shared module) to land, or build against the current converters/polaris/ structure for now?
   
   Sorry that PR #235 did not lead to much discussion. Also, the `common` module it introduced only contains the OSI models generated from `core-spec/osi-schema.json`. I think you can feel free to move forward with your idea.
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ossie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org