You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "andygrove (via GitHub)" <gi...@apache.org> on 2026/04/23 15:12:39 UTC
[I] Spark 4.0 sql_hive-1: HiveUDFDynamicLoadSuite fails to load UDF classes from hive-contrib [datafusion-comet]
andygrove opened a new issue, #4049:
URL: https://github.com/apache/datafusion-comet/issues/4049
## Describe the bug
With the `sql_hive-1` Spark SQL CI job re-enabled for Spark 4.0 (see #2946 / #4047), five tests in `org.apache.spark.sql.hive.HiveUDFDynamicLoadSuite` fail:
- `Spark should be able to run Hive UDF using jar regardless of current thread context classloader (UDF)`
- `Spark should be able to run Hive UDF using jar regardless of current thread context classloader (GENERIC_UDF)`
- `Spark should be able to run Hive UDF using jar regardless of current thread context classloader (GENERIC_UDAF)`
- `Spark should be able to run Hive UDF using jar regardless of current thread context classloader (UDAF)`
- `Spark should be able to run Hive UDF using jar regardless of current thread context classloader (GENERIC_UDTF)`
All five fail with:
```
org.apache.spark.sql.AnalysisException: [CANNOT_LOAD_FUNCTION_CLASS] Cannot load class
org.apache.hadoop.hive.contrib.udf.example.UDFExampleAdd2 when registering the function
`spark_catalog`.`default`.`udf_add2`, please make sure it is on the classpath. SQLSTATE: 46103
```
The same tests pass on Spark 3.4 and 3.5 under the same CI matrix. Only the Spark 4.0 sql_hive-1 job is affected.
The tests dynamically register Hive UDFs that live in `hive-contrib`; on Spark 4.0 that jar (or the class inside it) isn't resolvable on the session classpath of the Hive test harness.
## Reproduction
Run the `sql_hive-1` job for Spark 4.0.1 (equivalent to `build/sbt -Pspark-4.0 hive/testOnly * -- -l org.apache.spark.tags.ExtendedHiveTest -l org.apache.spark.tags.SlowHiveTest` from a prepared apache-spark checkout, as wired up in `.github/workflows/spark_sql_test.yml`).
Observed on CI run https://github.com/apache/datafusion-comet/actions/runs/24837941861/job/72704298074 (PR #4047).
## Expected behavior
All five HiveUDFDynamicLoadSuite variants should pass, matching the behavior on Spark 3.4/3.5.
## Additional context
These failures are unrelated to #2946 (hang/timeout, fixed in #4048). They remain as a blocker to re-enabling the `sql_hive-1` job for Spark 4.0. Likely candidates: `hive-contrib` classpath/packaging differences between Spark 3.x and 4.0 test harnesses, or a classloader isolation change in Spark 4.0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Spark 4.0 sql_hive-1: HiveUDFDynamicLoadSuite fails to load UDF classes from hive-contrib [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove closed issue #4049: Spark 4.0 sql_hive-1: HiveUDFDynamicLoadSuite fails to load UDF classes from hive-contrib
URL: https://github.com/apache/datafusion-comet/issues/4049
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org