You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "liuzqt (via GitHub)" <gi...@apache.org> on 2024/02/27 21:45:46 UTC

Re: [PR] [SPARK-43157][SQL] Clone InMemoryRelation cached plan to prevent cloned plan from referencing same objects [spark]

liuzqt commented on PR #40812:
URL: https://github.com/apache/spark/pull/40812#issuecomment-1967661413

   I'm revisiting this PR since it introduced a new issue https://issues.apache.org/jira/browse/SPARK-47177 I don't have a clear idea yet how to fix this, a few ideas out of my head:
   - maybe we just a coarse granularity lock in explain?
   - alternatively, make `innerChildren` a function: clone the initial plan, every time check for whether the original AQE plan is finalized (making the final flag atomic first, of course), if no: return the cloned initial plan, if it's finalized: clone the final plan and return that one. But still this won't be able to reflect the AQE plan in real time, in a concurrent situation, but at least we have initial version and final version.
   
   
   cc @cloud-fan to provide some feedback


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org