You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "karuppayya (via GitHub)" <gi...@apache.org> on 2026/03/19 21:03:20 UTC
[PR] Use thread context classloader for Iceberg class loading [datafusion-comet]
karuppayya opened a new pull request, #3738:
URL: https://github.com/apache/datafusion-comet/pull/3738
## Which issue does this PR close?
Closes https://github.com/apache/datafusion-comet/issues/3737
## Rationale for this change
Comet uses `Class.forName(className)` to reflectively load Iceberg classes.
The single-argument `Class.forName` uses the classloader of the **calling class**, not the thread context classloader.
When the Comet JAR is placed in Spark's `jars/` folder, it is loaded by the system classloader.
However, Iceberg classes may be bundled inside the user's application JAR(as uber jar), which is loaded by Spark's [application classloader](https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala#L939-L950) .
Since parent classloaders cannot see classes loaded by child classloaders, `Class.forName` from Comet's code fails with `ClassNotFoundException`.
([`SparkClassUtils.classForName`](https://github.com/apache/spark/blob/master/common/utils/src/main/scala/org/apache/spark/util/SparkClassUtils.scala#L37-L48) I guess does the same)
## What changes are included in this PR?
- Adds `IcebergReflection.loadClass()` helper that resolves classes via the thread context classloader (falling back to the default if null).
- Replaces all bare `Class.forName` calls in `IcebergReflection` and `CometIcebergNativeScan` with the new helper method.
## How are these changes tested?
Verified in a deployment where the Comet JAR is in Spark's `jars/` folder and Iceberg is bundled in the application uber JAR and ran few SQLs which involved reading from Iceberg.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: Use thread context classloader for Iceberg class loading [datafusion-comet]
Posted by "karuppayya (via GitHub)" <gi...@apache.org>.
karuppayya commented on PR #3738:
URL: https://github.com/apache/datafusion-comet/pull/3738#issuecomment-4093282839
cc: @parthchandra @andygrove @mbutrovich
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: Use thread context classloader for Iceberg class loading [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove merged PR #3738:
URL: https://github.com/apache/datafusion-comet/pull/3738
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: Use thread context classloader for Iceberg class loading [datafusion-comet]
Posted by "parthchandra (via GitHub)" <gi...@apache.org>.
parthchandra commented on code in PR #3738:
URL: https://github.com/apache/datafusion-comet/pull/3738#discussion_r2963107517
##########
spark/src/main/scala/org/apache/comet/iceberg/IcebergReflection.scala:
##########
@@ -77,6 +77,29 @@ object IcebergReflection extends Logging {
val UNKNOWN = "unknown"
}
+ /**
+ * Loads a class using the thread context classloader first, then falls back to the system
+ * classloader.
+ *
+ * @param className
+ * Fully qualified class name to load
+ * @return
+ * The loaded Class object
+ */
+ def loadClass(className: String): Class[_] = {
Review Comment:
If there is a class loading error, it would be useful to log which loader was used.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org