You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "MIchael Davies (JIRA)" <ji...@apache.org> on 2015/02/01 11:56:34 UTC
[jira] [Updated] (SPARK-5511) [SQL] Possible optimisations for
predicate pushdowns from Spark SQL to Parquet
[ https://issues.apache.org/jira/browse/SPARK-5511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
MIchael Davies updated SPARK-5511:
----------------------------------
Description:
The following changes could make predicate pushdown more effective under certain conditions, which are not uncommon.
1. Parquet predicate evaluation does not use dictionary compression information, furthermore it circumvents dictionary decoding optimisations -https://issues.apache.org/jira/browse/PARQUET-36. This means predicates are re-evaluated repeatedly for the same Strings, and also Binary->String conversions are repeated. This is a change purely on the Parquet side.
2. Support IN clauses in predicate pushdown. This requires changes to Parquet and then subsequently in Spark SQL.
was:
The following changes could make predicate pushdown more effective under certain conditions, which are not uncommon.
1. Parquet predicate evaluation does not use dictionary compression information, furthermore it circumvents dictionary decoding optimisations. This means predicates are re-evaluated repeatedly for the same Strings, and also Binary->String conversions are repeated. This is a change purely on the Parquet side.
2. Support IN clauses in predicate pushdown. This requires changes to Parquet and then subsequently in Spark SQL.
> [SQL] Possible optimisations for predicate pushdowns from Spark SQL to Parquet
> ------------------------------------------------------------------------------
>
> Key: SPARK-5511
> URL: https://issues.apache.org/jira/browse/SPARK-5511
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 1.2.0
> Reporter: MIchael Davies
> Priority: Minor
>
> The following changes could make predicate pushdown more effective under certain conditions, which are not uncommon.
> 1. Parquet predicate evaluation does not use dictionary compression information, furthermore it circumvents dictionary decoding optimisations -https://issues.apache.org/jira/browse/PARQUET-36. This means predicates are re-evaluated repeatedly for the same Strings, and also Binary->String conversions are repeated. This is a change purely on the Parquet side.
> 2. Support IN clauses in predicate pushdown. This requires changes to Parquet and then subsequently in Spark SQL.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org