[jira] [Created] (SPARK-32694) Pushdown cast to data sources

["Chao Sun (Jira)" <ji...@apache.org>]

Chao Sun created SPARK-32694:
--------------------------------

             Summary: Pushdown cast to data sources
                 Key: SPARK-32694
                 URL: https://issues.apache.org/jira/browse/SPARK-32694
             Project: Spark
          Issue Type: Improvement
          Components: SQL
    Affects Versions: 3.0.0
            Reporter: Chao Sun


Currently we don't support pushing down cast to data source (see [link|http://apache-spark-developers-list.1001551.n3.nabble.com/SparkSql-Casting-of-Predicate-Literals-tp29956p30035.html] for a discussion). For instance, in the following code snippet:

{code}
case class Person(name: String, age: Short)
Seq(Person("John", 32), Person("David", 25), Person("Mike", 18)).toDS().write.parquet("/tmp/person.parquet")
val personDS = spark.read.parquet("/tmp/person.parquet")
personDS.createOrReplaceTempView("person")
spark.sql("SELECT * FROM person where age < 30")
{code}

The predicate won't be pushed down to Parquet data source because in {{DataSourceStrategy}}, {{PushableColumnBase}} only handles a few limited cases such as {{Attribute}} and {{GetStructField}}. Potentially we can handle {{Cast}} here as well.




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org