You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Younes Naguib <Yo...@tritondigital.com> on 2015/10/16 19:20:36 UTC

Dynamic partition pruning

Hi all

I'm running sqls on spark 1.5.1 and using tables based on parquets.
My tables are not pruned when joined on partition columns.
Ex:
Select .... from tab where partcol=1 will prune on value 1
Select .... from tab join dim on (dim.partcol=tab.partcol) where dim.partcol=1 will scan all partitions.

Any ideas or workarounds?


Thanks,
Younes

RE: Dynamic partition pruning

Posted by Younes Naguib <Yo...@tritondigital.com>.

Done: SPARK-11150

Thanks

From: Xiao Li [mailto:gatorsmile@gmail.com]
Sent: October-16-15 8:21 PM
To: Younes Naguib
Cc: Michael Armbrust; user@spark.apache.org
Subject: Re: Dynamic partition pruning

Hi, Younes,

Maybe you can open a JIRA?

Thanks,

Xiao Li

2015-10-16 12:43 GMT-07:00 Younes Naguib <Yo...@tritondigital.com>>:
Thanks,
Do you have a Jira I can follow for this?

y

From: Michael Armbrust [mailto:michael@databricks.com<ma...@databricks.com>]
Sent: October-16-15 2:18 PM
To: Younes Naguib
Cc: user@spark.apache.org<ma...@spark.apache.org>
Subject: Re: Dynamic partition pruning

We don't support dynamic partition pruning yet.

On Fri, Oct 16, 2015 at 10:20 AM, Younes Naguib <Yo...@tritondigital.com>> wrote:
Hi all

I’m running sqls on spark 1.5.1 and using tables based on parquets.
My tables are not pruned when joined on partition columns.
Ex:
Select .... from tab where partcol=1 will prune on value 1
Select .... from tab join dim on (dim.partcol=tab.partcol) where dim.partcol=1 will scan all partitions.

Any ideas or workarounds?

Thanks,
Younes

Re: Dynamic partition pruning

Posted by Xiao Li <ga...@gmail.com>.

Hi, Younes,

Maybe you can open a JIRA?

Thanks,

Xiao Li

2015-10-16 12:43 GMT-07:00 Younes Naguib <Yo...@tritondigital.com>:

> Thanks,
>
> Do you have a Jira I can follow for this?
>
>
>
> y
>
>
>
> *From:* Michael Armbrust [mailto:michael@databricks.com]
> *Sent:* October-16-15 2:18 PM
> *To:* Younes Naguib
> *Cc:* user@spark.apache.org
> *Subject:* Re: Dynamic partition pruning
>
>
>
> We don't support dynamic partition pruning yet.
>
>
>
> On Fri, Oct 16, 2015 at 10:20 AM, Younes Naguib <
> Younes.Naguib@tritondigital.com> wrote:
>
> Hi all
>
>
>
> I’m running sqls on spark 1.5.1 and using tables based on parquets.
>
> My tables are not pruned when joined on partition columns.
>
> Ex:
>
> Select .... from tab where partcol=1 will prune on value 1
>
> Select .... from tab join dim on (dim.partcol=tab.partcol) where
> dim.partcol=1 will scan all partitions.
>
>
>
> Any ideas or workarounds?
>
>
>
>
>
> *Thanks,*
>
> *Younes*
>
>
>
>
>

RE: Dynamic partition pruning

Posted by Younes Naguib <Yo...@tritondigital.com>.

Thanks,
Do you have a Jira I can follow for this?

y

From: Michael Armbrust [mailto:michael@databricks.com]
Sent: October-16-15 2:18 PM
To: Younes Naguib
Cc: user@spark.apache.org
Subject: Re: Dynamic partition pruning

We don't support dynamic partition pruning yet.

On Fri, Oct 16, 2015 at 10:20 AM, Younes Naguib <Yo...@tritondigital.com>> wrote:
Hi all

I’m running sqls on spark 1.5.1 and using tables based on parquets.
My tables are not pruned when joined on partition columns.
Ex:
Select .... from tab where partcol=1 will prune on value 1
Select .... from tab join dim on (dim.partcol=tab.partcol) where dim.partcol=1 will scan all partitions.

Any ideas or workarounds?

Thanks,
Younes

Re: Dynamic partition pruning

Posted by Michael Armbrust <mi...@databricks.com>.

We don't support dynamic partition pruning yet.

On Fri, Oct 16, 2015 at 10:20 AM, Younes Naguib <
Younes.Naguib@tritondigital.com> wrote:

> Hi all
>
>
>
> I’m running sqls on spark 1.5.1 and using tables based on parquets.
>
> My tables are not pruned when joined on partition columns.
>
> Ex:
>
> Select .... from tab where partcol=1 will prune on value 1
>
> Select .... from tab join dim on (dim.partcol=tab.partcol) where
> dim.partcol=1 will scan all partitions.
>
>
>
> Any ideas or workarounds?
>
>
>
>
>
> *Thanks,*
>
> *Younes*
>
>
>