You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Marco Gaido (Jira)" <ji...@apache.org> on 2019/09/01 07:27:00 UTC
[jira] [Updated] (SPARK-28939) SQL configuration are not always
propagated
[ https://issues.apache.org/jira/browse/SPARK-28939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Marco Gaido updated SPARK-28939:
--------------------------------
Description:
The SQL configurations are propagated to executors in order to be effective.
Unfortunately, in some cases, we are missing to propagate them, making them un-effective.
The problem happens every time {{rdd}} or {{queryExecution.toRdd}} are used. And this is pretty frequent in the codebase.
Please notice that there are 2 parts of this issue:
- when a user directly uses those APIs
- when Spark invokes them (eg. throughout the ML lib and other usages or the {{describe}} method on the {{Dataset}} class)
was:
The SQL configurations are propagated to executors in order to be effective.
Unfortunately, in some cases, we are missing to propagate them, making them uneffective.
For an example, please see the {{describe}} method on the {{Dataset}} class.
> SQL configuration are not always propagated
> -------------------------------------------
>
> Key: SPARK-28939
> URL: https://issues.apache.org/jira/browse/SPARK-28939
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.4.4
> Reporter: Marco Gaido
> Priority: Major
>
> The SQL configurations are propagated to executors in order to be effective.
> Unfortunately, in some cases, we are missing to propagate them, making them un-effective.
> The problem happens every time {{rdd}} or {{queryExecution.toRdd}} are used. And this is pretty frequent in the codebase.
> Please notice that there are 2 parts of this issue:
> - when a user directly uses those APIs
> - when Spark invokes them (eg. throughout the ML lib and other usages or the {{describe}} method on the {{Dataset}} class)
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org