You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "sundapeng (via GitHub)" <gi...@apache.org> on 2026/09/11 17:38:17 UTC
[PR] feat(datafusion): add SHOW, ADD and DROP PARTITION for catalog-managed format tables [paimon-rust]
sundapeng opened a new pull request, #816:
URL: https://github.com/apache/paimon-rust/pull/816
### Purpose
Child of #591, which stays open as a draft that shows the full feature; its remaining changes reach `main` through child PRs.
A Format Table loaded from a REST catalog with `metastore.partitioned-table=true` has its partitions registered in the catalog, and since #814 a scan reads exactly those registrations. DataFusion could not list or change them. This PR adds `SHOW PARTITIONS` and `ALTER TABLE ... ADD / DROP PARTITION` for such tables.
**Depends on:** nothing. It is based on `main`, where #811, #813, #812 and #814 have merged. `MSCK REPAIR TABLE` (<MSCK_PR>) and `ANALYZE TABLE` (#815) build on this PR.
### Brief change log
- `SHOW PARTITIONS table [PARTITION (...)]` lists the registered partitions in escaped `key=value/...` form, sorted. Values are read with the column type: a DATE registered as epoch days prints as `dt=2026-07-22`, `month=01` and `month=1` both print as `month=1`, and the default partition prints as `null`. The optional `PARTITION (...)` clause filters by any subset of partition values, compared the same way.
- `ALTER TABLE table ADD [IF NOT EXISTS] PARTITION (...) [PARTITION (...)]` registers complete specifications with the catalog, then creates their directories. `LOCATION` is refused.
- `ALTER TABLE table DROP [IF EXISTS] PARTITION (...)` follows Java `resolveFormatTablePartitionsForDrop`:
- several specifications per statement; a specification that fixes only some keys, leading or not, expands to every registered partition it matches, and may match none;
- a complete specification that matches nothing fails the whole statement unless `IF EXISTS` is given;
- values are compared as the catalog holds them, so `month = 1` does not select a partition registered as `month=01`;
- a statement made only of complete specifications looks them up through `list-by-names`; one with a partial specification reads the registry once;
- registrations are removed first and directories deleted afterwards; a partition at a custom location (`options.path`) is only unregistered and its data is left in place;
- it cannot be combined with other `ALTER TABLE` operations.
- Partition literals are read the way Java reads partition strings. A SQL literal becomes the string Spark would hand to Paimon (quoted strings and `DATE '...'` as written, booleans as `true`/`false`, numbers in canonical form, `NULL` as the default partition), and `parse_format_partition_value` reads it with the column type, as `TypeUtils.castFromString` does. So `month = '01'` and `month = 01` both name `month=1`, a BOOLEAN accepts `t/true/y/yes/1` and `f/false/n/no/0`, and a DATE accepts `yyyy-MM-dd`. An invalid value fails with its column named.
- `Catalog::drop_partitions`, with a default that returns `Unsupported`. `RESTCatalog` sends it in batches of 1000 and refuses tables without catalog-managed partitions.
- `FormatTablePartitionPaths`, `format_partition_value` and `parse_format_partition_value` become public in `paimon::table`.
### Tests
- `crates/integrations/datafusion/tests/rest_format_partition_sql.rs`:
- `test_partition_commands_update_rest_metadata_and_directories`: ADD of several partitions, SHOW with and without a filter, DROP, and their directories;
- `test_partition_literals`: DATE, INT, BOOLEAN and VARCHAR literals, including Java boolean spellings, unquoted, zero-padded and negative numbers and NULL, with the names and directories they produce, and invalid literals rejected with the column named;
- `test_drop_partition_specifications`: partial and non-leading specifications, several specifications, a missing complete specification with and without `IF EXISTS`, an empty partial match, a failing specification leaving the statement unapplied, and the refusal to combine with other operations, each checking the remaining registrations and directories;
- `test_drop_partition_matches_values_as_the_catalog_holds_them`: SHOW reads `month=01` and `month=1` with the column type, while DROP compares the values as registered;
- `test_drop_partition_looks_up_complete_specifications_by_name`;
- `test_drop_partition_leaves_a_custom_location_in_place`;
- a compile-time check that the future of `SQLContext::sql` stays `Send`.
- `crates/paimon/tests/rest_catalog_test.rs`: `test_rest_catalog_skips_empty_and_batches_drop_partitions`, `test_rest_catalog_refuses_to_drop_partitions_it_does_not_manage` and `test_rest_catalog_drop_partitions_maps_missing_table`.
- `sql_context` unit test `test_drop_if_exists_partition_does_not_ignore_missing_table`.
Commands run on the top commit:
```
cargo fmt --all -- --check
cargo clippy --locked --all-targets --workspace --features fulltext,vortex -- -D warnings
cargo test --locked -p paimon --all-targets --features fulltext,vortex
cargo test --locked --no-fail-fast -p paimon-datafusion --all-targets
cargo test --locked -p paimon-rest-server --all-targets
```
fmt and clippy pass. `paimon`: 2960 tests pass. `paimon-rest-server`: 10 tests pass. `paimon-datafusion`: 771 pass, including the 6 tests in `rest_format_partition_sql.rs`, and 39 fail. The failures are the 39 tests that also fail by name on `main` in this environment, because they read fixture tables provisioned by `make docker-up` (two also need the lumina native library).
### Additions without a direct Java counterpart
- `SQLContext` turns SQL partition literals into partition strings itself (`partition_literal_to_string`): in Java, Spark resolves the partition spec before Paimon sees it, and DataFusion has no such step for these statements. The strings are then read by `parse_format_partition_value`, the counterpart of `TypeUtils.castFromString`.
- `SQLContext` parses `SHOW PARTITIONS` itself, since sqlparser 0.62 has no such statement.
- `RESTCatalog::drop_partitions` refuses tables without catalog-managed partitions, where Java commits `truncatePartitions`: turning string specs into typed values for every Paimon partition type is not available yet, and the endpoint alone would report success without touching data.
- The compile-time `sql_future_is_send` test keeps the future of `SQLContext::sql` `Send`, which Rust callers need for `tokio::spawn` and `#[async_trait]`.
### API and Format
- New `Catalog::drop_partitions`, with a default implementation that returns `Unsupported`.
- `paimon::table::FormatTablePartitionPaths`, `format_partition_value` and `parse_format_partition_value` become public.
The storage format is unchanged.
### Documentation
`docs/src/sql.md` gains a Format Table Partitions section: the `metastore.partitioned-table` prerequisite, `SHOW` / `ADD` / `DROP PARTITION`, how partition values are read, the ordering guarantees of ADD and DROP, and how a partition at a custom location is treated. The SQL support scope lists the new statements.
### Scope and limitations
- Applies only to partitioned internal REST Format Tables with `metastore.partitioned-table=true` and a non-`engine` implementation.
- Several `DROP PARTITION` specifications are written as repeated clauses, `DROP PARTITION (...), DROP PARTITION (...)`; sqlparser 0.62 cannot parse the Hive form `DROP PARTITION (...), (...)`.
- Reading a partition from its custom location is not implemented yet; such partitions fail closed.
- A TIME partition value must be given as an integer. Partition columns typed DECIMAL, TIMESTAMP, FLOAT, DOUBLE or BINARY remain unsupported on this path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(datafusion): add SHOW, ADD and DROP PARTITION for catalog-managed format tables [paimon-rust]
Posted by "sundapeng (via GitHub)" <gi...@apache.org>.
sundapeng commented on code in PR #816:
URL: https://github.com/apache/paimon-rust/pull/816#discussion_r3995377637
##########
crates/integrations/datafusion/src/sql_context.rs:
##########
@@ -3033,6 +3450,222 @@ fn is_table_not_exist(e: &paimon::Error) -> bool {
matches!(e, paimon::Error::TableNotExist { .. })
}
+/// Whether the catalog registered a partition at a location of its own rather than under the
+/// table directory.
+fn has_custom_location(partition: &paimon::spec::Partition) -> bool {
+ partition
+ .options
+ .as_ref()
+ .is_some_and(|options| options.contains_key("path"))
+}
+
+fn ensure_catalog_managed_format_table(table: &paimon::Table, operation: &str) -> DFResult<()> {
+ if table.schema().partition_keys().is_empty() {
+ return Err(DataFusionError::Plan(format!(
+ "{operation} requires a partitioned table, but {} is not partitioned",
+ table.identifier().full_name()
+ )));
+ }
+ if !table.has_catalog_managed_partitions() {
+ return Err(DataFusionError::Plan(format!(
+ "{operation} is supported only for catalog-managed partitions on an internal Format Table loaded from REST Catalog; table {} does not have this configuration",
+ table.identifier().full_name()
+ )));
+ }
+ Ok(())
+}
+
+fn parse_format_partition_spec(
+ exprs: &[SqlExpr],
+ table: &paimon::Table,
+ require_complete: bool,
+ enable_ident_normalization: bool,
+) -> DFResult<HashMap<String, String>> {
+ let fields = table
+ .schema()
+ .fields()
+ .iter()
+ .map(|field| (field.name(), field))
+ .collect::<HashMap<_, _>>();
+ let partition_keys = table.schema().partition_keys();
+ let options = CoreOptions::new(table.schema().options());
+ let mut spec = HashMap::with_capacity(exprs.len());
+
+ for expr in exprs {
+ let (column, literal) = partition_assignment(expr, enable_ident_normalization)?;
+ if !partition_keys.contains(&column) {
+ return Err(DataFusionError::Plan(format!(
+ "Column '{column}' is not a partition column"
+ )));
+ }
+ if spec.contains_key(&column) {
+ return Err(DataFusionError::Plan(format!(
+ "Duplicate partition column '{column}'"
+ )));
+ }
+ let field = fields.get(column.as_str()).ok_or_else(|| {
+ DataFusionError::Plan(format!("Column '{column}' not found in table schema"))
+ })?;
+ let data_type = field.data_type();
+ let value = match partition_literal_to_string(literal)? {
+ None => options.partition_default_name().to_string(),
+ // The literal is read with the column type the way a registration or directory is,
+ // which is how Java casts partition strings (`TypeUtils.castFromString`), and written
+ // back in the spelling ADD PARTITION registers.
+ Some(text) => {
+ let text = match data_type {
+ PaimonDataType::Char(_) | PaimonDataType::VarChar(_) => text,
+ _ => text.trim().to_string(),
+ };
+ parse_format_partition_value(&text, data_type)
+ .and_then(|datum| {
+ format_partition_value(
+ &datum,
+ data_type,
+ options.partition_default_name(),
+ options.legacy_partition_name(),
+ )
Review Comment:
Agreed, fixed in c0fe18c.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(datafusion): add SHOW, ADD and DROP PARTITION for catalog-managed format tables [paimon-rust]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #816:
URL: https://github.com/apache/paimon-rust/pull/816
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(datafusion): add SHOW, ADD and DROP PARTITION for catalog-managed format tables [paimon-rust]
Posted by "sundapeng (via GitHub)" <gi...@apache.org>.
sundapeng commented on PR #816:
URL: https://github.com/apache/paimon-rust/pull/816#issuecomment-5646611300
Agreed, moved to `crates/integrations/datafusion/src/format_partition_ddl.rs` in f18d972.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(datafusion): add SHOW, ADD and DROP PARTITION for catalog-managed format tables [paimon-rust]
Posted by "QuakeWang (via GitHub)" <gi...@apache.org>.
QuakeWang commented on code in PR #816:
URL: https://github.com/apache/paimon-rust/pull/816#discussion_r3995341689
##########
crates/integrations/datafusion/src/sql_context.rs:
##########
@@ -3033,6 +3450,222 @@ fn is_table_not_exist(e: &paimon::Error) -> bool {
matches!(e, paimon::Error::TableNotExist { .. })
}
+/// Whether the catalog registered a partition at a location of its own rather than under the
+/// table directory.
+fn has_custom_location(partition: &paimon::spec::Partition) -> bool {
+ partition
+ .options
+ .as_ref()
+ .is_some_and(|options| options.contains_key("path"))
+}
+
+fn ensure_catalog_managed_format_table(table: &paimon::Table, operation: &str) -> DFResult<()> {
+ if table.schema().partition_keys().is_empty() {
+ return Err(DataFusionError::Plan(format!(
+ "{operation} requires a partitioned table, but {} is not partitioned",
+ table.identifier().full_name()
+ )));
+ }
+ if !table.has_catalog_managed_partitions() {
+ return Err(DataFusionError::Plan(format!(
+ "{operation} is supported only for catalog-managed partitions on an internal Format Table loaded from REST Catalog; table {} does not have this configuration",
+ table.identifier().full_name()
+ )));
+ }
+ Ok(())
+}
+
+fn parse_format_partition_spec(
+ exprs: &[SqlExpr],
+ table: &paimon::Table,
+ require_complete: bool,
+ enable_ident_normalization: bool,
+) -> DFResult<HashMap<String, String>> {
+ let fields = table
+ .schema()
+ .fields()
+ .iter()
+ .map(|field| (field.name(), field))
+ .collect::<HashMap<_, _>>();
+ let partition_keys = table.schema().partition_keys();
+ let options = CoreOptions::new(table.schema().options());
+ let mut spec = HashMap::with_capacity(exprs.len());
+
+ for expr in exprs {
+ let (column, literal) = partition_assignment(expr, enable_ident_normalization)?;
+ if !partition_keys.contains(&column) {
+ return Err(DataFusionError::Plan(format!(
+ "Column '{column}' is not a partition column"
+ )));
+ }
+ if spec.contains_key(&column) {
+ return Err(DataFusionError::Plan(format!(
+ "Duplicate partition column '{column}'"
+ )));
+ }
+ let field = fields.get(column.as_str()).ok_or_else(|| {
+ DataFusionError::Plan(format!("Column '{column}' not found in table schema"))
+ })?;
+ let data_type = field.data_type();
+ let value = match partition_literal_to_string(literal)? {
+ None => options.partition_default_name().to_string(),
+ // The literal is read with the column type the way a registration or directory is,
+ // which is how Java casts partition strings (`TypeUtils.castFromString`), and written
+ // back in the spelling ADD PARTITION registers.
+ Some(text) => {
+ let text = match data_type {
+ PaimonDataType::Char(_) | PaimonDataType::VarChar(_) => text,
+ _ => text.trim().to_string(),
+ };
+ parse_format_partition_value(&text, data_type)
+ .and_then(|datum| {
+ format_partition_value(
+ &datum,
+ data_type,
+ options.partition_default_name(),
+ options.legacy_partition_name(),
+ )
Review Comment:
`format_partition_value` maps empty and whitespace-only strings to the default partition name. As a result, `DROP PARTITION (dt = '')` deletes the NULL partition, including its files—I reproduced this locally. Please reject these literals before formatting and add a regression test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org