You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@drill.apache.org by "Adam Gilmore (JIRA)" <ji...@apache.org> on 2015/03/23 07:30:10 UTC
[jira] [Updated] (DRILL-2517) Partition pruning still reading files
during planning
[ https://issues.apache.org/jira/browse/DRILL-2517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Adam Gilmore updated DRILL-2517:
--------------------------------
Description:
Partition pruning still tries to read Parquet files during the planning stage even though they don't match the partition filter.
For example, if there were an invalid Parquet file in a directory that should not be queried:
{code}
0: jdbc:drill:zk=local> select sum(price) from dfs.tmp.purchases where dir0 = 1;
Query failed: IllegalArgumentException: file:/tmp/purchases/4/0_0_0.parquet is not a Parquet file (too small)
{code}
The reason is that the partition pruning happens after the Parquet plugin tries to read the footer of each file.
Ideally, partition pruning would happen first before the format plugin gets involved.
was:
Partition pruning still tries to read Parquet files during the planning stage even though they don't match the partition filter.
For example:
{code}
0: jdbc:drill:zk=local> select sum(price) from dfs.tmp.purchases where dir0 = 1;
Query failed: IllegalArgumentException: file:/tmp/purchases/4/0_0_0.parquet is not a Parquet file (too small)
{code}
The reason is that the partition pruning happens after the Parquet plugin tries to read the footer of each file.
Ideally, partition pruning would happen first before the format plugin gets involved.
> Partition pruning still reading files during planning
> -----------------------------------------------------
>
> Key: DRILL-2517
> URL: https://issues.apache.org/jira/browse/DRILL-2517
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
> Affects Versions: 0.7.0, 0.8.0
> Reporter: Adam Gilmore
> Assignee: Jinfeng Ni
>
> Partition pruning still tries to read Parquet files during the planning stage even though they don't match the partition filter.
> For example, if there were an invalid Parquet file in a directory that should not be queried:
> {code}
> 0: jdbc:drill:zk=local> select sum(price) from dfs.tmp.purchases where dir0 = 1;
> Query failed: IllegalArgumentException: file:/tmp/purchases/4/0_0_0.parquet is not a Parquet file (too small)
> {code}
> The reason is that the partition pruning happens after the Parquet plugin tries to read the footer of each file.
> Ideally, partition pruning would happen first before the format plugin gets involved.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)