You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "huaxingao (via GitHub)" <gi...@apache.org> on 2026/01/02 20:37:04 UTC
[PR] SQL UDF support Stage 1 [iceberg]
huaxingao opened a new pull request, #14954:
URL: https://github.com/apache/iceberg/pull/14954
This PR adds Stage 1 support for Iceberg-managed SQL scalar UDFs in Spark 4.1, backed by a REST service. Spark can list functions and execute them without session-scoped temporary registration by using an analyzer rewrite.
Scope is intentionally read-only: list + load + execute (no CREATE/REPLACE/DROP).
named arguments , default values, UDTF, CRUD will be supported in the following up stages
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] SQL UDF support Stage 1 [iceberg]
Posted by "flyrain (via GitHub)" <gi...@apache.org>.
flyrain commented on code in PR #14954:
URL: https://github.com/apache/iceberg/pull/14954#discussion_r2743892300
##########
spark/v4.1/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestRestScalarFunctionCatalog.java:
##########
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark.extensions;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.node.ObjectNode;
+import java.util.Arrays;
+import org.apache.iceberg.CatalogProperties;
+import org.apache.iceberg.ParameterizedTestExtension;
+import org.apache.iceberg.Parameters;
+import org.apache.iceberg.catalog.Namespace;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.spark.SparkCatalogConfig;
+import org.apache.spark.sql.connector.catalog.FunctionCatalog;
+import org.apache.spark.sql.connector.catalog.Identifier;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.TestTemplate;
+import org.junit.jupiter.api.extension.ExtendWith;
+
+/**
+ * End-to-end test for REST-backed scalar SQL UDFs executed via Iceberg Spark extensions.
+ *
+ * <p>Stage 1: scalar only. UDTF/table functions will be added in a follow-up (stage 2).
+ */
+@ExtendWith(ParameterizedTestExtension.class)
+public class TestRestScalarFunctionCatalog extends ExtensionsTestBase {
+
+ private static final String FUNCTION_NAMESPACE = "udf_ns";
+
+ // Catalog-backed (production) REST functions are only supported for the REST catalog type.
+ @Parameters(name = "catalogName = {0}, implementation = {1}, config = {2}")
+ protected static Object[][] parameters() {
+ return new Object[][] {
+ {
+ SparkCatalogConfig.REST.catalogName(),
+ SparkCatalogConfig.REST.implementation(),
+ ImmutableMap.builder()
+ .putAll(SparkCatalogConfig.REST.properties())
+ .put(CatalogProperties.URI, restCatalog.properties().get(CatalogProperties.URI))
+ .build()
+ }
+ };
+ }
+
+ private static final ObjectMapper MAPPER = new ObjectMapper();
+
+ private static final String ADD_ONE_SPEC =
+ "{"
+ + "\"function-uuid\":\"123\","
+ + "\"format-version\":1,"
+ + "\"doc\":\"Add one UDF\","
+ + "\"parameter-names\":[{\"name\":\"x\"}],"
+ + "\"definitions\":[{"
+ + " \"definition-id\":\"(int)\","
+ + " \"parameters\":[{\"type\":\"int\"}],"
Review Comment:
Tests for nested type(`list`, `struct`, `map`) are needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] SQL UDF support Stage 1 [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #14954:
URL: https://github.com/apache/iceberg/pull/14954#issuecomment-3978702114
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] SQL UDF support Stage 1 [iceberg]
Posted by "flyrain (via GitHub)" <gi...@apache.org>.
flyrain commented on code in PR #14954:
URL: https://github.com/apache/iceberg/pull/14954#discussion_r2743889247
##########
spark/v4.1/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestRestScalarFunctionCatalog.java:
##########
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark.extensions;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.node.ObjectNode;
+import java.util.Arrays;
+import org.apache.iceberg.CatalogProperties;
+import org.apache.iceberg.ParameterizedTestExtension;
+import org.apache.iceberg.Parameters;
+import org.apache.iceberg.catalog.Namespace;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.spark.SparkCatalogConfig;
+import org.apache.spark.sql.connector.catalog.FunctionCatalog;
+import org.apache.spark.sql.connector.catalog.Identifier;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.TestTemplate;
+import org.junit.jupiter.api.extension.ExtendWith;
+
+/**
+ * End-to-end test for REST-backed scalar SQL UDFs executed via Iceberg Spark extensions.
+ *
+ * <p>Stage 1: scalar only. UDTF/table functions will be added in a follow-up (stage 2).
+ */
+@ExtendWith(ParameterizedTestExtension.class)
+public class TestRestScalarFunctionCatalog extends ExtensionsTestBase {
+
+ private static final String FUNCTION_NAMESPACE = "udf_ns";
+
+ // Catalog-backed (production) REST functions are only supported for the REST catalog type.
+ @Parameters(name = "catalogName = {0}, implementation = {1}, config = {2}")
+ protected static Object[][] parameters() {
+ return new Object[][] {
+ {
+ SparkCatalogConfig.REST.catalogName(),
+ SparkCatalogConfig.REST.implementation(),
+ ImmutableMap.builder()
+ .putAll(SparkCatalogConfig.REST.properties())
+ .put(CatalogProperties.URI, restCatalog.properties().get(CatalogProperties.URI))
+ .build()
+ }
+ };
+ }
+
+ private static final ObjectMapper MAPPER = new ObjectMapper();
+
+ private static final String ADD_ONE_SPEC =
+ "{"
+ + "\"function-uuid\":\"123\","
+ + "\"format-version\":1,"
+ + "\"doc\":\"Add one UDF\","
+ + "\"parameter-names\":[{\"name\":\"x\"}],"
+ + "\"definitions\":[{"
+ + " \"definition-id\":\"(int)\","
Review Comment:
We are trying to remove parens from the definition id, `(int)` -> `int`. We can change it until the UDF spec is finalized.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] SQL UDF support Stage 1 [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #14954:
URL: https://github.com/apache/iceberg/pull/14954#issuecomment-4020365900
This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] SQL UDF support Stage 1 [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #14954: SQL UDF support Stage 1
URL: https://github.com/apache/iceberg/pull/14954
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] SQL UDF support Stage 1 [iceberg]
Posted by "flyrain (via GitHub)" <gi...@apache.org>.
flyrain commented on code in PR #14954:
URL: https://github.com/apache/iceberg/pull/14954#discussion_r2743893877
##########
core/src/main/java/org/apache/iceberg/udf/UdfSpecResolver.java:
##########
@@ -0,0 +1,535 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.udf;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.node.ArrayNode;
+import com.fasterxml.jackson.databind.node.ObjectNode;
+import java.util.Iterator;
+import java.util.List;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+
+/** Engine-agnostic resolver for Iceberg UDF spec JSON. */
+public final class UdfSpecResolver {
+
+ private static final ObjectMapper MAPPER = new ObjectMapper();
+
+ private UdfSpecResolver() {}
+
+ private static final String FUNCTION_UUID = "function-uuid";
+ private static final String FORMAT_VERSION = "format-version";
+ private static final String PARAMETER_NAMES = "parameter-names";
+ private static final String DEFINITIONS = "definitions";
+ private static final String DEFINITION_LOG = "definition-log";
+ private static final String SECURE = "secure";
+ private static final String DOC = "doc";
+ private static final String LOCATION = "location";
+ private static final String PROPERTIES = "properties";
+
+ private static final String DEFINITION_ID = "definition-id";
+ private static final String PARAMETERS = "parameters";
+ private static final String RETURN_TYPE = "return-type";
+ private static final String FUNCTION_TYPE = "function-type";
+ private static final String CURRENT_VERSION_ID = "current-version-id";
+ private static final String VERSIONS = "versions";
+ private static final String NULLABLE_RETURN = "nullable-return";
+
+ private static final String VERSION_ID = "version-id";
+ private static final String TIMESTAMP_MS = "timestamp-ms";
+ private static final String REPRESENTATIONS = "representations";
+ private static final String DETERMINISTIC = "deterministic";
+ private static final String ON_NULL_INPUT = "on-null-input";
+ private static final String DEFINITION_VERSIONS = "definition-versions";
+
+ private static final String TYPE = "type";
+ private static final String DEFAULT = "default";
+ private static final String DIALECT = "dialect";
+ private static final String BODY = "body";
+
+ private static final class ParameterName {
+ private final String name;
+ private final String doc;
+
+ private ParameterName(String name, String doc) {
+ this.name = name;
+ this.doc = doc;
+ }
+ }
+
+ /**
+ * Resolve all definitions' current versions for a dialect, returning one ResolvedSpec per
+ * definition (overload).
+ */
+ @SuppressWarnings("checkstyle:CyclomaticComplexity")
+ public static List<ResolvedSpec> resolveAll(String udfMetadataJson, String dialect) {
+ try {
+ ObjectNode root = (ObjectNode) MAPPER.readTree(udfMetadataJson);
+ validateRoot(root);
+
+ boolean secure = root.has(SECURE) && root.get(SECURE).asBoolean(false);
+ if (secure) {
+ // Engines must not expose secure function bodies via inspection; Stage 1 implementations
+ // do not support secure functions.
+ throw new IllegalArgumentException("Secure UDFs are not supported");
+ }
+
+ ArrayNode definitions = (ArrayNode) root.get(DEFINITIONS);
+ List<ParameterName> globalParameterNames = parseGlobalParameterNames(root);
Review Comment:
We have removed the globalParameterNames from the spec.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org