You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "David Milicevic (Jira)" <ji...@apache.org> on 2024/06/08 08:29:00 UTC
[jira] [Updated] (SPARK-48530) [M0] Support for local variables
[ https://issues.apache.org/jira/browse/SPARK-48530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
David Milicevic updated SPARK-48530:
------------------------------------
Description:
At the moment, variables in SQL scripts are creating session variables. We don't want this, we want variables to be considered as local (within the block/compound).
To achieve this, we probably need to wait for labels support. Once we have it, we can prepend variable names with labels to make distinction between variables with the same name and only then reuse session variables mechanism to save values with such composed names.
If the block/compound doesn't have label, we should generate it automatically (GUID or something similar).
Also, variables cannot recover from script failures - if a script fails, variables won’t be dropped. If we intend to reuse session variables for local variables, we should think about how to fix this.
was:
At the moment, variables in SQL scripts are creating session variables. We don't want this, we want variables to be considered as local (within the block/compound).
To achieve this, we probably need to wait for labels support. Once we have it, we can prepend variable names with labels to make distinction between variables with the same name and only then reuse session variables mechanism to save values with such composed names.
If the block/compound doesn't have label, we should generate it automatically (GUID or something similar).
> [M0] Support for local variables
> --------------------------------
>
> Key: SPARK-48530
> URL: https://issues.apache.org/jira/browse/SPARK-48530
> Project: Spark
> Issue Type: Sub-task
> Components: Spark Core
> Affects Versions: 4.0.0
> Reporter: David Milicevic
> Priority: Major
>
> At the moment, variables in SQL scripts are creating session variables. We don't want this, we want variables to be considered as local (within the block/compound).
>
> To achieve this, we probably need to wait for labels support. Once we have it, we can prepend variable names with labels to make distinction between variables with the same name and only then reuse session variables mechanism to save values with such composed names.
> If the block/compound doesn't have label, we should generate it automatically (GUID or something similar).
>
> Also, variables cannot recover from script failures - if a script fails, variables won’t be dropped. If we intend to reuse session variables for local variables, we should think about how to fix this.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org