You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "Zouxxyy (via GitHub)" <gi...@apache.org> on 2026/10/12 08:50:31 UTC
[PR] [spark] Support REST catalog Hive UDTFs in lateral views [paimon]
Zouxxyy opened a new pull request, #10510:
URL: https://github.com/apache/paimon/pull/10510
### Purpose
A Hive UDTF registered with `CREATE FUNCTION` in a REST catalog works in a `SELECT` list, but `LATERAL VIEW` fails with `ROUTINE_NOT_FOUND`: Spark represents the call as `UnresolvedGenerator`, which Paimon did not rewrite.
Resolve these generators through the existing Paimon V1 function registry after their arguments resolve, and retain Spark's generator type validation. Bind `database.function` to the current catalog, including when the database and a configured catalog share a name. Preserve built-in and temporary function precedence, and document Hive UDTF invocation.
### Tests
- `PaimonV1FunctionTest`: 17 passed on Spark 3.5.8 (Scala 2.12, JDK 8).
- `PaimonV1FunctionTest`: 17 passed on Spark 4.1.2 (Scala 2.13, JDK 17).
- Five new cases cover persistent registration and cold lookup, qualified names, Paimon table input, empty/null input and `OUTER`, a fresh session, CTEs, dependent generators, name precedence, and rejection of scalar functions in `LATERAL VIEW`.
- `spotless:apply`, `spotless:check`, and `git diff --check` passed.
- Spark 4.1 reactor `package -DskipTests` passed without `fast-build`, with Checkstyle, Spotless, and Enforcer enabled.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [spark] Support REST catalog Hive UDTFs in lateral views [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #10510:
URL: https://github.com/apache/paimon/pull/10510#issuecomment-6159228166
+1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [spark] Support REST catalog Hive UDTFs in lateral views [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #10510:
URL: https://github.com/apache/paimon/pull/10510#discussion_r4253525326
##########
paimon-spark/paimon-spark-common/src/main/scala/org/apache/spark/sql/catalyst/parser/extensions/RewritePaimonFunctionCommands.scala:
##########
@@ -123,28 +125,91 @@ case class RewritePaimonFunctionCommands(spark: SparkSession) extends Rule[Logic
u,
alias => transformPaimonV1Function(alias).asInstanceOf[SubqueryAlias])
case l: LogicalPlan =>
- l.transformExpressionsWithPruning(_.containsAnyPattern(UNRESOLVED_FUNCTION)) {
+ l.transformExpressionsWithPruning(_.containsAnyPattern(UNRESOLVED_FUNCTION, GENERATOR)) {
Review Comment:
[P2] Visit expression subquery plans when rewriting generators
This traversal does not enter `SubqueryExpression.plan`, so the new handler misses ordinary scalar/EXISTS queries containing a REST UDTF in LATERAL VIEW. After `CREATE FUNCTION review_udtf AS 'org.apache.hadoop.hive.ql.udf.generic.GenericUDTFPosExplode'`, I reproduced `ROUTINE_NOT_FOUND` with:
```sql
SELECT (SELECT count(*)
FROM VALUES (array(10,20)) AS t(items)
LATERAL VIEW review_udtf(items) u AS position,item) AS n;
```
The equivalent `posexplode` query returns 2, and an EXISTS version fails for the same reason. Spark's recursive subquery analyzer does not rerun this parser rewrite; the remaining native `UnresolvedGenerator` goes to SessionCatalog rather than Paimon's private registry. This leaves the PR's new LATERAL VIEW support incomplete for count/existence query composition. Recurse through expression subquery plans and replace them with their rewritten plans (while retaining CTE handling), and add scalar/EXISTS integration cases.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [spark] Support REST catalog Hive UDTFs in lateral views [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #10510:
URL: https://github.com/apache/paimon/pull/10510
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org