You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "szehon-ho (via GitHub)" <gi...@apache.org> on 2026/06/08 18:45:35 UTC
[PR] Spec: Add optional specific-name to UDF definition model [iceberg]
szehon-ho opened a new pull request, #16727:
URL: https://github.com/apache/iceberg/pull/16727
## Summary
Adds an optional `specific-name` field to the UDF function definition model in the [UDF spec](https://github.com/apache/iceberg/blob/main/format/udf-spec.md), analogous to the SQL standard's routine *specific name*.
Resolves #16700
## Motivation
In the SQL standard, a routine has two distinct identifiers:
- a **routine name** — the user-facing function name, which may be shared across overloads, and
- a **specific name** — a schema-unique name that identifies a single overload unambiguously.
The specific name exists precisely because referencing an overload by `name + signature` is awkward, so the standard lets a user assign a stable, human-readable handle that engines can use for statements like `DROP SPECIFIC FUNCTION`.
Today the UDF model only has `definition-id`, which is *derived from the canonical parameter-type tuple*. It uniquely identifies an overload, but it is an opaque, signature-bound string rather than a user-chosen name. This change adds `specific-name` as an optional convenience alongside `definition-id`, giving engines a stable name to agree on without overloading the meaning of `definition-id`.
### SQL-standard example
A user specifies the specific name at creation time with the `SPECIFIC` clause:
\`\`\`sql
CREATE FUNCTION add_one(x INT)
RETURNS INT
SPECIFIC add_one_int
RETURN x + 1;
CREATE FUNCTION add_one(x FLOAT)
RETURNS FLOAT
SPECIFIC add_one_float
RETURN x + 1.0;
\`\`\`
and can then reference a single overload by that name, independent of its signature:
\`\`\`sql
DROP SPECIFIC FUNCTION add_one_int;
\`\`\`
This maps directly onto the new field: the two overloads above serialize with `"specific-name": "add_one_int"` and `"specific-name": "add_one_float"` respectively (see the updated Appendix A example).
## Design notes
- `specific-name` is **optional** and **backward compatible** — `definition-id` is left unchanged, so existing metadata stays valid and readers that ignore the field still work.
- It is for **identification only** and **must not** be used for overload resolution; resolution remains based on the definition's `parameters`.
- When present, `specific-name` must be unique among all definitions in the UDF metadata, and must not collide with any other definition's `definition-id`, so a single string references at most one definition.
- When absent, `definition-id` continues to serve as the definition's identifier.
## Changes
- New optional `specific-name` row in the Definition fields table.
- New **Specific Name** subsection defining its semantics and uniqueness rules.
- Updated Appendix A (overloaded `add_one`) to illustrate the field.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spec: Add optional specific-name to UDF definition model [iceberg]
Posted by "szehon-ho (via GitHub)" <gi...@apache.org>.
szehon-ho commented on code in PR #16727:
URL: https://github.com/apache/iceberg/pull/16727#discussion_r3392519053
##########
format/udf-spec.md:
##########
@@ -131,6 +132,18 @@ Examples of complete definition-id signatures:
* `int,string` – two parameters: int and string
* `int,list<int>,struct<id:int,name:string>` – three parameters: an int, a list and a struct
+#### Specific Name
+The `specific-name` is an optional, user-assignable identifier for a single definition, analogous to the SQL standard's
+routine *specific name*. It provides a stable handle for a definition that is independent of its signature, which engines
+may use to reference a specific overload unambiguously (e.g., for SQL statements such as `DROP SPECIFIC FUNCTION`).
+
+* `specific-name` is for identification only. It **must not** be used for overload resolution; resolution is based on the
+ definition's `parameters` (see [Function Call Convention and Resolution in Engines](#function-call-convention-and-resolution-in-engines)).
+* When present, `specific-name` **must** be unique among all definitions within the UDF metadata.
Review Comment:
Good catch. We considered specifying exact case-sensitive equality here, but decided **not** to define case-sensitivity (or any folding) in the spec, for consistency with how the rest of Iceberg's catalog handles names.
In the REST catalog spec, namespace levels and table/view names are plain opaque `string`s with no normalization rules — and the only `case-sensitive` flag in that spec is explicitly scoped to scan field matching, not to identifiers. Identifier folding (the `Add_One` vs `add_one` question, regular vs. delimited identifiers, Spark lowercasing vs. an uppercasing engine) is applied by the engine *before* the name is written, exactly as it is for table and namespace names today.
If the UDF spec mandated exact case-sensitive equality, it would be specifying *more* about name comparison than the catalog does for table names, and it still wouldn't resolve cross-engine folding disagreements, which genuinely live at the engine layer. Within a single metadata file the rule is deterministic — uniqueness is over the literal stored `definition-id` strings — so we leave folding to the engine rather than introduce a UDF-specific identifier-comparison rule that diverges from the catalog.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spec: Add optional specific-name to UDF definition model [iceberg]
Posted by "szehon-ho (via GitHub)" <gi...@apache.org>.
szehon-ho commented on code in PR #16727:
URL: https://github.com/apache/iceberg/pull/16727#discussion_r3478447166
##########
format/udf-spec.md:
##########
@@ -220,6 +228,7 @@ RETURN x + 1.0;
"definitions": [
{
"definition-id": "int",
+ "specific-name": "add_one_int",
Review Comment:
yea this is a good point. actually Spark does not yet either, but hopefully it can one day
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spec: Add optional specific-name to UDF definition model [iceberg]
Posted by "szehon-ho (via GitHub)" <gi...@apache.org>.
szehon-ho commented on code in PR #16727:
URL: https://github.com/apache/iceberg/pull/16727#discussion_r3478458787
##########
format/udf-spec.md:
##########
@@ -220,6 +228,7 @@ RETURN x + 1.0;
"definitions": [
{
"definition-id": "int",
+ "specific-name": "add_one_int",
Review Comment:
Removed the `specific-name` fields from the Appendix A JSON example so it stays consistent with the Trino/Spark SQL above (which don't use a `SPECIFIC` clause yet). Thanks for catching this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spec: Add optional specific-name to UDF definition model [iceberg]
Posted by "wombatu-kun (via GitHub)" <gi...@apache.org>.
wombatu-kun commented on code in PR #16727:
URL: https://github.com/apache/iceberg/pull/16727#discussion_r3385136984
##########
format/udf-spec.md:
##########
@@ -131,6 +132,18 @@ Examples of complete definition-id signatures:
* `int,string` – two parameters: int and string
* `int,list<int>,struct<id:int,name:string>` – three parameters: an int, a list and a struct
+#### Specific Name
+The `specific-name` is an optional, user-assignable identifier for a single definition, analogous to the SQL standard's
+routine *specific name*. It provides a stable handle for a definition that is independent of its signature, which engines
+may use to reference a specific overload unambiguously (e.g., for SQL statements such as `DROP SPECIFIC FUNCTION`).
+
+* `specific-name` is for identification only. It **must not** be used for overload resolution; resolution is based on the
+ definition's `parameters` (see [Function Call Convention and Resolution in Engines](#function-call-convention-and-resolution-in-engines)).
+* When present, `specific-name` **must** be unique among all definitions within the UDF metadata.
Review Comment:
These uniqueness and collision MUST rules depend on a comparison basis the spec leaves unstated. Unlike `definition-id`, which is a canonical lowercase string, `specific-name` is user-assignable free text, so two engines could disagree on whether `Add_One` collides with `add_one`. Consider stating the comparison semantics (e.g. exact, case-sensitive string equality) so the rule is interoperable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spec: Allow user-specified definition-id in UDF model [iceberg]
Posted by "szehon-ho (via GitHub)" <gi...@apache.org>.
szehon-ho commented on PR #16727:
URL: https://github.com/apache/iceberg/pull/16727#issuecomment-4676137976
## Design options considered for the "specific name"
There were two ways to give an overload a stable, user-facing handle:
**Option A — a separate `specific-name` field.** Keep `definition-id` strictly derived from the signature and add a new optional `specific-name` field. Pros: maps 1:1 onto the SQL standard's two-name model and preserves the verifiable invariant that `definition-id == canonical(parameters)`. Cons: introduces a second identifier and a second reference key (e.g. `definition-log` would have to choose which one to record).
**Option B — make `definition-id` optionally user-specifiable (chosen).** Let `definition-id` default to the canonical parameter-type tuple but allow a user-specified value. Pros: no new field, reuses the existing `definition-log` reference mechanism, smallest surface area. Cons: `definition-id` is no longer always the canonical signature, so it can't be recomputed/validated from `parameters` alone, and uniqueness must be stated explicitly rather than implied by the signature.
### Decision
We went with **Option B**. Keeping a single identifier (rather than two) and not adding a second reference key to `definition-log` outweighed the loss of the "ID always equals the canonical signature" invariant, which we recovered by stating the uniqueness rule explicitly and keeping the canonical form as the default. Overload resolution continues to use `parameters`, so making `definition-id` user-specifiable does not affect resolution.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spec: Add optional specific-name to UDF definition model [iceberg]
Posted by "szehon-ho (via GitHub)" <gi...@apache.org>.
szehon-ho commented on PR #16727:
URL: https://github.com/apache/iceberg/pull/16727#issuecomment-4652308675
## Alternative considered: making `definition-id` user-specifiable (not chosen)
While designing this, we considered an alternative to adding a separate field: **allow `definition-id` itself to be optionally user-specified**, so it could double as the specific name.
### What Option B would look like
Instead of deriving `definition-id` strictly from the canonical parameter-type tuple, it would *default* to that canonical form but be overridable with any user-chosen, unique string. The signature used for overload resolution and the one-definition-per-signature rule would then be based solely on the `parameters` list, and `definition-id` would become an opaque identity token (which could carry the human-readable "specific name").
### Why we didn't choose it
- **It breaks a useful, checkable invariant.** Today `definition-id == canonical(parameters)`, so a reader can recompute the ID from the signature and validate it (some readers even parse it to recover types). Making it user-overridable removes that property and requires a *separate* uniqueness check on `parameters` that currently comes for free.
- **Backward-compatibility risk.** Existing consumers may assume `definition-id` equals the signature. Allowing arbitrary overrides could silently break them. Option A leaves `definition-id` completely untouched, so it's purely additive.
- **It re-conflates two concepts.** The whole motivation (#16700) is that the SQL standard keeps *signature* (for resolution) and *specific name* (for identification) separate. Option B reduces the field count but folds both roles back into one field — the exact conflation we're trying to address. A self-documenting, signature-derived `definition-id` plus an explicit `specific-name` keeps the two roles cleanly separated and matches the standard's model directly.
- **Footgun potential.** A user could set `definition-id: "int"` on a `float` overload, producing an ID that looks like a signature but isn't. The canonical-by-default form is self-documenting; a free-form override is not.
Net: Option A (this PR) is fully additive, preserves the verifiable canonical `definition-id`, and is the closest match to the SQL standard's two-name model, so we went with it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spec: Allow user-specified definition-id in UDF model [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #16727:
URL: https://github.com/apache/iceberg/pull/16727#discussion_r3399170051
##########
format/udf-spec.md:
##########
@@ -117,8 +117,14 @@ following fields required. Any other fields must be ignored.
e.g., `{ "type": "struct", "fields": [ { "name": "id", "type": "int" }, { "name": "name", "type": "string" } ] }`
#### Definition ID
-The `definition-id` is a canonical string derived from the parameter types, formatted as a comma-separated list with no
-spaces. Each type uses the following string representation:
+The `definition-id` is a string that uniquely identifies a definition within the UDF metadata. It may be a user-specified
Review Comment:
I think it would be fine to add an optional `specific-name` but I don't see a reason to change how `definition-id` works. That is derived from the parameters and I don't see a reason to change it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spec: Add optional specific-name to UDF definition model [iceberg]
Posted by "flyrain (via GitHub)" <gi...@apache.org>.
flyrain commented on code in PR #16727:
URL: https://github.com/apache/iceberg/pull/16727#discussion_r3432324891
##########
format/udf-spec.md:
##########
@@ -131,6 +132,12 @@ Examples of complete definition-id signatures:
* `int,string` – two parameters: int and string
* `int,list<int>,struct<id:int,name:string>` – three parameters: an int, a list and a struct
+#### Specific Name
+The `specific-name` is an optional, user-assignable name for a single definition, analogous to the SQL standard's
+routine *specific name*. It provides a stable, human-readable handle for a definition that is independent of its
+signature (e.g., for SQL statements such as `DROP SPECIFIC FUNCTION`).
Review Comment:
Looks like it's part of the sql standard, https://ronsavage.github.io/SQL/sql-99.bnf.html.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spec: Add optional specific-name to UDF definition model [iceberg]
Posted by "szehon-ho (via GitHub)" <gi...@apache.org>.
szehon-ho merged PR #16727:
URL: https://github.com/apache/iceberg/pull/16727
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spec: Add optional specific-name to UDF definition model [iceberg]
Posted by "ebyhr (via GitHub)" <gi...@apache.org>.
ebyhr commented on code in PR #16727:
URL: https://github.com/apache/iceberg/pull/16727#discussion_r3478422821
##########
format/udf-spec.md:
##########
@@ -220,6 +228,7 @@ RETURN x + 1.0;
"definitions": [
{
"definition-id": "int",
+ "specific-name": "add_one_int",
Review Comment:
nit: This JSON is located after the Trino and Spark SQL examples. Since it doesn't include a SPECIFIC clause, readers may wonder where "specific-name": "add_one_int" comes from.
Does Spark support the SPECIFIC clause? If it does, we could add it to the SQL example.
It's unsupported as far as I checked https://spark.apache.org/docs/latest/sql-ref-syntax-ddl-create-sql-function.html though.
I know Trino doesn't support it. I'll bring this issue to the attention of the Trino community.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spec: Add optional specific-name to UDF definition model [iceberg]
Posted by "szehon-ho (via GitHub)" <gi...@apache.org>.
szehon-ho commented on PR #16727:
URL: https://github.com/apache/iceberg/pull/16727#issuecomment-4860507498
Merged, thanks all for review!
Vote thread for reference: https://lists.apache.org/thread/75xs4trcgcpog8b2zoxpknf8bq6vfk4c
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spec: Add optional specific-name to UDF definition model [iceberg]
Posted by "szehon-ho (via GitHub)" <gi...@apache.org>.
szehon-ho commented on PR #16727:
URL: https://github.com/apache/iceberg/pull/16727#issuecomment-4736257112
Reverted the switch to a user-specified `definition-id` and went back to a separate optional `specific-name` field, per @rdblue's feedback to keep `definition-id` derived from the parameters. The PR description has been updated to match.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spec: Add optional specific-name to UDF definition model [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #16727:
URL: https://github.com/apache/iceberg/pull/16727#discussion_r3477782351
##########
format/udf-spec.md:
##########
@@ -131,6 +132,12 @@ Examples of complete definition-id signatures:
* `int,string` – two parameters: int and string
* `int,list<int>,struct<id:int,name:string>` – three parameters: an int, a list and a struct
+#### Specific Name
+The `specific-name` is an optional, user-assignable name for a single definition, analogous to the SQL standard's
Review Comment:
Style: Add a newline between section headings and the text.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org