You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Eron Wright (JIRA)" <ji...@apache.org> on 2015/07/02 17:25:04 UTC
[jira] [Created] (SPARK-8794) Column pruning isn't applied to
certain transformed DataFrames
Eron Wright created SPARK-8794:
-----------------------------------
Summary: Column pruning isn't applied to certain transformed DataFrames
Key: SPARK-8794
URL: https://issues.apache.org/jira/browse/SPARK-8794
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 1.4.0
Reporter: Eron Wright
I observe that certain transformations (e.g. sample) on DataFrame cause the underlying relation's support for column pruning to be disregarded in subsequent queries.
I encountered this issue while using an ML pipeline with a typical dataset of (label, features). For my particular data source (which implements PrunedScan), the 'features' column is expensive to compute while the 'label' column is cheap. The first stage of the pipeline - StringIndexer - operates only on the label and so should be quick. Yet I found that the 'features' column would be materialized. Upon investigation, the issue occurs when the dataset is split into train/test with sampling. The sampling transformation causes the pruning optimization to be lost.
See this gist for a sample program demonstrating the issue:
[https://gist.github.com/EronWright/cb5fb9af46fd810194f8]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org