You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Chao Sun (Jira)" <ji...@apache.org> on 2020/08/24 20:50:00 UTC
[jira] [Updated] (SPARK-32694) Pushdown cast to data sources
[ https://issues.apache.org/jira/browse/SPARK-32694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chao Sun updated SPARK-32694:
-----------------------------
Description:
Currently we don't support pushing down cast to data source (see [link|http://apache-spark-developers-list.1001551.n3.nabble.com/SparkSql-Casting-of-Predicate-Literals-tp29956p30035.html] for a discussion). For instance, in the following code snippet:
{code}
> case class Person(name: String, age: Short)
> Seq(Person("John", 32), Person("David", 25), Person("Mike", 18)).toDS().write.parquet("/tmp/person.parquet")
> val personDS = spark.read.parquet("/tmp/person.parquet")
> personDS.createOrReplaceTempView("person")
> spark.sql("SELECT * FROM person where age < 30")
{code}
The predicate won't be pushed down to Parquet data source because in {{DataSourceStrategy}}, {{PushableColumnBase}} only handles a few limited cases such as {{Attribute}} and {{GetStructField}}. Potentially we can handle {{Cast}} here as well.
was:
Currently we don't support pushing down cast to data source (see [link|http://apache-spark-developers-list.1001551.n3.nabble.com/SparkSql-Casting-of-Predicate-Literals-tp29956p30035.html] for a discussion). For instance, in the following code snippet:
{code}
case class Person(name: String, age: Short)
Seq(Person("John", 32), Person("David", 25), Person("Mike", 18)).toDS().write.parquet("/tmp/person.parquet")
val personDS = spark.read.parquet("/tmp/person.parquet")
personDS.createOrReplaceTempView("person")
spark.sql("SELECT * FROM person where age < 30")
{code}
The predicate won't be pushed down to Parquet data source because in {{DataSourceStrategy}}, {{PushableColumnBase}} only handles a few limited cases such as {{Attribute}} and {{GetStructField}}. Potentially we can handle {{Cast}} here as well.
> Pushdown cast to data sources
> -----------------------------
>
> Key: SPARK-32694
> URL: https://issues.apache.org/jira/browse/SPARK-32694
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 3.0.0
> Reporter: Chao Sun
> Priority: Major
>
> Currently we don't support pushing down cast to data source (see [link|http://apache-spark-developers-list.1001551.n3.nabble.com/SparkSql-Casting-of-Predicate-Literals-tp29956p30035.html] for a discussion). For instance, in the following code snippet:
> {code}
> > case class Person(name: String, age: Short)
> > Seq(Person("John", 32), Person("David", 25), Person("Mike", 18)).toDS().write.parquet("/tmp/person.parquet")
> > val personDS = spark.read.parquet("/tmp/person.parquet")
> > personDS.createOrReplaceTempView("person")
> > spark.sql("SELECT * FROM person where age < 30")
> {code}
> The predicate won't be pushed down to Parquet data source because in {{DataSourceStrategy}}, {{PushableColumnBase}} only handles a few limited cases such as {{Attribute}} and {{GetStructField}}. Potentially we can handle {{Cast}} here as well.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org