Re: [I] AbstractMethodError: CometBroadcastExchangeExec missing sparkContext() from BroadcastExchangeLike [datafusion-comet]

["mbutrovich (via GitHub)" <gi...@apache.org>]

mbutrovich commented on issue #4318:
URL: https://github.com/apache/datafusion-comet/issues/4318#issuecomment-4672478040

   Thanks for the follow-up @dor-bernstein, this is really helpful.
   
   One thing stands out: `BroadcastExchangeLike` is byte-identical across Spark 3.5.6, 3.5.7, and 3.5.8, and Comet 0.16 builds its spark-3.5 artifact against 3.5.8, so a matching `comet-spark-spark3.5` jar on stock Apache Spark 3.5.6 should not hit this. The `$$super$sparkContext` accessor in the error is the classic signature of `CometBroadcastExchangeExec` having been compiled against a different Spark ABI than it runs on, which usually means either a mismatched Comet artifact or a non-stock Spark build.
   
   A couple of questions to narrow it down:
   
   1. Are you running on a vendor distribution like EMR, rather than stock Apache Spark? Vendor builds fork and recompile Spark internals, so their `BroadcastExchangeLike` ABI can differ from Apache 3.5.6 even when the version string matches.
   2. Which exact Comet artifact are you using (for example `comet-spark-spark3.5_2.12`), and how is it placed on the classpath?
   
   That should tell us whether this is an artifact mismatch or a vendor-Spark ABI difference. Either way it looks separate from the Iceberg reflection issue that #3895 fixed.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org