[jira] [Updated] (SPARK-48530) [M1] Support for local variables

["ASF GitHub Bot (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-48530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

ASF GitHub Bot updated SPARK-48530:
-----------------------------------
    Labels: pull-request-available  (was: )

> [M1] Support for local variables
> --------------------------------
>
>                 Key: SPARK-48530
>                 URL: https://issues.apache.org/jira/browse/SPARK-48530
>             Project: Spark
>          Issue Type: Sub-task
>          Components: Spark Core
>    Affects Versions: 4.0.0
>            Reporter: David Milicevic
>            Priority: Major
>              Labels: pull-request-available
>
> At the moment, variables in SQL scripts are creating session variables. We don't want this, we want variables to be considered as local (within the block/compound).
>  
> -To achieve this, we probably need to wait for labels support. Once we have it, we can prepend variable names with labels to make distinction between variables with the same name and only then reuse session variables mechanism to save values with such composed names.-
> -If the block/compound doesn't have label, we should generate it automatically (GUID or something similar).-
>  
> Labels support is done, so we can use them now for local variables - they need to be able to be referenced by <labelName>.<varName>.
> We cannot reuse session variables for this - we need to implement local variables per script, with scoping and shadowing.
>  
> Session variables need to be able to be accessed by `system.session.<varName>` at all times from the script though.
>  
> -Also, variables cannot recover from script failures - if a script fails, variables won’t be dropped. If we intend to reuse session variables for local variables, we should think about how to fix this.- We won't be using session variables, so this shouldn't be a problem, but need to have it in mind because wrong implementation can cause the same problem.
>  
> Use [Session Variables PR|https://github.com/apache/spark/pull/40474/files] as a reference in which places variable changes need to happen.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org