[REVIEW REQUEST] PR #2395 - Fix ClassCastException when deserializing runtime bloom filter subquery (AURON #2386)

[Chloe Xie <ch...@gmail.com>]

Hi Auron devs,

I would like to ask for a review of PR #2395, which fixes issue #2386:

  https://github.com/apache/auron/pull/2395

Summary of the problem: NativeConverters.deserializeExpression uses a plain
ObjectInputStream. Its default resolveClass resolves classes through
VM.latestUserDefinedLoader(), which picks a class loader from the live call
stack rather than the thread context class loader. When Auron is supplied
via
spark.jars (so it is defined by Spark's MutableURLClassLoader), the selected
loader often cannot see Auron classes. The ScalarSubquery expression graph
injected by Spark's runtime bloom filter optimizer is then only partially
resolved, and an un-readResolve'd DefaultSerializationProxy ends up
assigned to
RDD.dependencies_, raising a ClassCastException on Spark 4.1 / Scala 2.13.

The fix pins class resolution to an explicit loader (the context/Spark
loader,
falling back to the loader that defined Auron), which is what Spark's own
JavaDeserializationStream does. It is not reused directly because it is
private[spark] and Auron builds against eight Spark versions.

The PR also adds a TPC-DS CI job with the runtime bloom filter enabled and
the
jar supplied only through --jars. The job fails on the commit preceding the
fix
(10/10 shards) and passes with it. Local runs on Spark 4.1.2/4.1.3 at sf=1
and
sf=10 show the same before/after behaviour, with results validated against
vanilla Spark.

One open question for reviewers: the CI job and the fix are in the same PR.
I am happy to split the CI job into a separate PR if that is preferred.

Any feedback would be much appreciated.

Thanks,
Xiaoyan Xie