You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Josh Rosen (JIRA)" <ji...@apache.org> on 2015/05/26 04:45:17 UTC
[jira] [Updated] (SPARK-7858) DataSourceStrategy.createPhysicalRDD
should use output schema when performing row conversions, not relation
schema
[ https://issues.apache.org/jira/browse/SPARK-7858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Josh Rosen updated SPARK-7858:
------------------------------
Description: In {{DataSourceStrategy.createPhysicalRDD}}, we use the relation schema as the target schema for converting incoming rows into Catalyst rows. However, we should be using the output schema instead, since our scan might return a subset of the relation's columns. (was: In {{DataSourceStrategy.createPhysicalRDD}}, we use the relation schema as the target schema for converting incoming rows into Catalyst rows. However, we should be using the output schema instead, since our scan might be returning fewer columns due to partition pruning.)
> DataSourceStrategy.createPhysicalRDD should use output schema when performing row conversions, not relation schema
> ------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-7858
> URL: https://issues.apache.org/jira/browse/SPARK-7858
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 1.4.0
> Reporter: Josh Rosen
> Assignee: Josh Rosen
>
> In {{DataSourceStrategy.createPhysicalRDD}}, we use the relation schema as the target schema for converting incoming rows into Catalyst rows. However, we should be using the output schema instead, since our scan might return a subset of the relation's columns.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org