[jira] [Created] (SPARK-17489) Improve filtering for bucketed tables

["Shuai Lin (JIRA)" <ji...@apache.org>]

Shuai Lin created SPARK-17489:
---------------------------------

             Summary: Improve filtering for bucketed tables
                 Key: SPARK-17489
                 URL: https://issues.apache.org/jira/browse/SPARK-17489
             Project: Spark
          Issue Type: Improvement
          Components: SQL
            Reporter: Shuai Lin


Datasource allows creation of bucketed tables, can we optimize the query planning when there is a filter on the bucketed column?

For example:

{code}
select * from bucked_table where bucketed_col = "foo"
{code}

Given the above query, spark should only load the bucket files corresponding to the bucket files of value "foo".

But the current implementation does load all the files. Here is a small program to demonstrate.

{code}
# bin/spark-shell --master="local[2]"

case class Foo(name: String, age: Int)
spark.createDataFrame(Seq(
  Foo("aaa", 1),
  Foo("aaa", 2), 
  Foo("bbb", 3), 
  Foo("bbb", 4)))
  .write
  .format("json")
  .mode("overwrite")
  .bucketBy(2, "name")
  .saveAsTable("foo")

spark.sql("select * from foo where name = 'aaa'").show()

{code}

Then use sysdig to capture the file read events:

{code}
$ sudo sysdig -A -p "*%evt.time %evt.buffer" "fd.name contains spark-warehouse" and "evt.buffer contains bbb"  

05:36:59.430426611 
{\"name\":\"bbb\",\"age\":3}
{\"name\":\"bbb\",\"age\":4}
{code}

Sysdig shows the bucket files that obviously doesn't match the filter (name = "aaa") are also read by spark.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org