You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "xiangfu0 (via GitHub)" <gi...@apache.org> on 2026/04/16 23:37:43 UTC
[PR] Add SQL DDL support for CREATE/DROP TABLE and SHOW CREATE TABLE [pinot]
xiangfu0 opened a new pull request, #18241:
URL: https://github.com/apache/pinot/pull/18241
## Summary
- **Parser**: Extends the Calcite FMPP/JavaCC grammar with `CREATE TABLE`, `DROP TABLE`, `SHOW TABLES`, and `SHOW CREATE TABLE` productions; adds 6 AST node classes (`SqlPinotCreateTable`, `SqlPinotDropTable`, `SqlPinotShowTables`, `SqlPinotShowCreateTable`, `SqlPinotColumnDeclaration`, `SqlPinotProperty`); new DDL keywords added as non-reserved so they can still be used as identifiers in queries.
- **New `pinot-sql-ddl` module**: `DdlCompiler` translates SQL DDL into `Schema` + `TableConfig`; `PropertyMapping` routes `PROPERTIES` entries (promoted scalars → builder fields; `stream.*`/`realtime.*` → `IndexingConfig.streamConfigs`; `task.<type>.<key>` → `TableTaskConfig`; JSON-blob fallback for complex nested configs; remainder → `TableCustomConfig` for forward-compatibility); `CanonicalDdlEmitter` + `PropertyExtractor` + `SchemaEmitter` implement the reverse direction (`SHOW CREATE TABLE`) producing a deterministic idempotent DDL string; `SqlIdentifiers` quotes column names that conflict with DDL/SQL/data-type keywords.
- **Controller endpoint**: `POST /sql/ddl` with optional `?dryRun=true`; handles CREATE (with schema+table rollback on partial failure), DROP (pre-authorized existence check to prevent probing), SHOW TABLES (DB-scoped), and SHOW CREATE TABLE.
## Design notes
- Stream and minion-task config schemas are intentionally free-format — no grammar changes are needed when Kafka or Minion config evolves.
- Custom-config keys that shadow promoted scalars, JSON-blob keys, or prefix-routed paths (`task.*`, `stream.*`, `realtime.*`) are rejected at emission time to prevent silent round-trip corruption.
- Authorization is performed after database-name translation on every code path to prevent cross-DB privilege escalation.
- Long-tail `IndexingConfig` / `SegmentsValidationAndRetentionConfig` fields not yet emitted are documented in `PropertyExtractor` Javadoc and deferred to a follow-up slice.
## Test plan
- [x] 26 parser tests (`PinotDdlParserTest`) — all syntax variants, `IF NOT EXISTS`, `IF EXISTS`, `SHOW CREATE TABLE`, keyword-as-identifier regression
- [x] 27 compiler tests (`DdlCompilerTest`) — all `PropertyMapping` routing rules, validation, negative cases, `DEFAULT` quoting, `realtime.*` routing
- [x] 14 emitter golden-output tests (`CanonicalDdlEmitterTest`) — exact string assertions lock canonical form; shadow-rejection tests for exact-match and prefix-routed keys
- [x] 11 round-trip idempotency tests (`RoundTripTest`) — `emit → parse → compile → emit` is idempotent across all routing rules; data-type-keyword column names; kitchen-sink promoted-scalars regression
- [x] 14 controller integration tests (`PinotDdlRestletResourceTest`) — create/dry-run/IF NOT EXISTS/409, drop/404/IF EXISTS, SHOW TABLES, SHOW CREATE TABLE, DB-qualified DROP regression, parse-error and semantic-error 400s
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Add SQL DDL support for CREATE/DROP TABLE and SHOW CREATE TABLE [pinot]
Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 commented on PR #18241:
URL: https://github.com/apache/pinot/pull/18241#issuecomment-4467378640
Opened docs PR pinot-contrib/pinot-docs#807 to document the new SQL table DDL endpoint () and its Querying & SQL / controller API docs surfaces.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Add SQL DDL support for CREATE/DROP TABLE and SHOW CREATE TABLE [pinot]
Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 merged PR #18241:
URL: https://github.com/apache/pinot/pull/18241
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Add SQL DDL support for CREATE/DROP TABLE and SHOW CREATE TABLE [pinot]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18241:
URL: https://github.com/apache/pinot/pull/18241#issuecomment-4264342944
## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/18241?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `42.75362%` with `79 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 55.27%. Comparing base ([`77ecc4c`](https://app.codecov.io/gh/apache/pinot/commit/77ecc4cd0c8562e0b4e98492508c85cb62284ef5?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`f507283`](https://app.codecov.io/gh/apache/pinot/commit/f507283578c4cd67c4c5a78b54994dff1ea8be8d?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
| [Files with missing lines](https://app.codecov.io/gh/apache/pinot/pull/18241?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [.../pinot/sql/parsers/parser/SqlPinotCreateTable.java](https://app.codecov.io/gh/apache/pinot/pull/18241?src=pr&el=tree&filepath=pinot-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fsql%2Fparsers%2Fparser%2FSqlPinotCreateTable.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcWwvcGFyc2Vycy9wYXJzZXIvU3FsUGlub3RDcmVhdGVUYWJsZS5qYXZh) | 35.13% | [24 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/18241?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [.../sql/parsers/parser/SqlPinotColumnDeclaration.java](https://app.codecov.io/gh/apache/pinot/pull/18241?src=pr&el=tree&filepath=pinot-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fsql%2Fparsers%2Fparser%2FSqlPinotColumnDeclaration.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcWwvcGFyc2Vycy9wYXJzZXIvU3FsUGlub3RDb2x1bW5EZWNsYXJhdGlvbi5qYXZh) | 42.85% | [20 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/18241?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...he/pinot/sql/parsers/parser/SqlPinotDropTable.java](https://app.codecov.io/gh/apache/pinot/pull/18241?src=pr&el=tree&filepath=pinot-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fsql%2Fparsers%2Fparser%2FSqlPinotDropTable.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcWwvcGFyc2Vycy9wYXJzZXIvU3FsUGlub3REcm9wVGFibGUuamF2YQ==) | 47.36% | [10 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/18241?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...che/pinot/sql/parsers/parser/SqlPinotProperty.java](https://app.codecov.io/gh/apache/pinot/pull/18241?src=pr&el=tree&filepath=pinot-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fsql%2Fparsers%2Fparser%2FSqlPinotProperty.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcWwvcGFyc2Vycy9wYXJzZXIvU3FsUGlub3RQcm9wZXJ0eS5qYXZh) | 46.66% | [8 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/18241?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ot/sql/parsers/parser/SqlPinotShowCreateTable.java](https://app.codecov.io/gh/apache/pinot/pull/18241?src=pr&el=tree&filepath=pinot-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fsql%2Fparsers%2Fparser%2FSqlPinotShowCreateTable.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcWwvcGFyc2Vycy9wYXJzZXIvU3FsUGlub3RTaG93Q3JlYXRlVGFibGUuamF2YQ==) | 46.66% | [8 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/18241?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...e/pinot/sql/parsers/parser/SqlPinotShowTables.java](https://app.codecov.io/gh/apache/pinot/pull/18241?src=pr&el=tree&filepath=pinot-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fsql%2Fparsers%2Fparser%2FSqlPinotShowTables.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcWwvcGFyc2Vycy9wYXJzZXIvU3FsUGlub3RTaG93VGFibGVzLmphdmE=) | 41.66% | [7 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/18241?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...org/apache/pinot/sql/parsers/CalciteSqlParser.java](https://app.codecov.io/gh/apache/pinot/pull/18241?src=pr&el=tree&filepath=pinot-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fsql%2Fparsers%2FCalciteSqlParser.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcWwvcGFyc2Vycy9DYWxjaXRlU3FsUGFyc2VyLmphdmE=) | 60.00% | [1 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/pinot/pull/18241?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
> :exclamation: There is a different number of reports uploaded between BASE (77ecc4c) and HEAD (f507283). Click for more details.
>
> <details><summary>HEAD has 36 uploads less than BASE</summary>
>
>| Flag | BASE (77ecc4c) | HEAD (f507283) |
>|------|------|------|
>|java-21|5|1|
>|unittests1|2|1|
>|unittests|4|1|
>|temurin|10|1|
>|java-11|5|0|
>|unittests2|2|0|
>|integration|6|0|
>|integration2|2|0|
>|integration1|2|0|
>|custom-integration1|2|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18241 +/- ##
============================================
- Coverage 63.40% 55.27% -8.13%
+ Complexity 1627 837 -790
============================================
Files 3243 2554 -689
Lines 197162 146134 -51028
Branches 30495 23487 -7008
============================================
- Hits 125009 80779 -44230
+ Misses 62151 58383 -3768
+ Partials 10002 6972 -3030
```
| [Flag](https://app.codecov.io/gh/apache/pinot/pull/18241/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [custom-integration1](https://app.codecov.io/gh/apache/pinot/pull/18241/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration](https://app.codecov.io/gh/apache/pinot/pull/18241/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration1](https://app.codecov.io/gh/apache/pinot/pull/18241/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration2](https://app.codecov.io/gh/apache/pinot/pull/18241/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [java-11](https://app.codecov.io/gh/apache/pinot/pull/18241/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [java-21](https://app.codecov.io/gh/apache/pinot/pull/18241/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `55.27% <42.75%> (-8.10%)` | :arrow_down: |
| [temurin](https://app.codecov.io/gh/apache/pinot/pull/18241/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `55.27% <42.75%> (-8.13%)` | :arrow_down: |
| [unittests](https://app.codecov.io/gh/apache/pinot/pull/18241/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `55.27% <42.75%> (-8.13%)` | :arrow_down: |
| [unittests1](https://app.codecov.io/gh/apache/pinot/pull/18241/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `55.27% <42.75%> (-0.07%)` | :arrow_down: |
| [unittests2](https://app.codecov.io/gh/apache/pinot/pull/18241/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/pinot/pull/18241?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Add SQL DDL support for CREATE/DROP TABLE and SHOW CREATE TABLE [pinot]
Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 commented on PR #18241:
URL: https://github.com/apache/pinot/pull/18241#issuecomment-4467378888
Opened docs PR pinot-contrib/pinot-docs#807 to document the new SQL table DDL endpoint (`POST /sql/ddl`) and its Querying & SQL / controller API docs surfaces.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org