[jira] [Updated] (SPARK-45136) Improve ClosureCleaner to support closures defined in Ammonite REPL

["ASF GitHub Bot (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-45136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

ASF GitHub Bot updated SPARK-45136:
-----------------------------------
    Labels: pull-request-available  (was: )

> Improve ClosureCleaner to support closures defined in Ammonite REPL
> -------------------------------------------------------------------
>
>                 Key: SPARK-45136
>                 URL: https://issues.apache.org/jira/browse/SPARK-45136
>             Project: Spark
>          Issue Type: Improvement
>          Components: Connect
>    Affects Versions: 4.0.0, 3.5.1
>            Reporter: Vsevolod Stepanov
>            Priority: Major
>              Labels: pull-request-available
>
> ConnectRepl uses Ammonite REPL with  CodeClassWrapper to run Scala code. It means that each code cell is wrapped into a separate object. If there are multiple variables defined in the same cell / code block it will lead to capturing extra variables, increasing serialized UDF payload size or making it non-serializable.
> For example, this code
> {code:java}
> // cell 1 
> {
>   val x = 100
>   val y = new NonSerializable
> }
> // cell 2
> spark.range(10).map(i => i + x).agg(sum("value")).collect(){code}
> will fail because lambda will capture both `x` and `y` as they're defined in the same wrapper object



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org