[jira] [Created] (SPARK-16196) Optimize in-memory scan performance using ColumnarBatches

["Andrew Or (JIRA)" <ji...@apache.org>]

Andrew Or created SPARK-16196:
---------------------------------

             Summary: Optimize in-memory scan performance using ColumnarBatches
                 Key: SPARK-16196
                 URL: https://issues.apache.org/jira/browse/SPARK-16196
             Project: Spark
          Issue Type: Bug
          Components: SQL
    Affects Versions: 2.0.0
            Reporter: Andrew Or
            Assignee: Andrew Or


A simple benchmark such as the following reveals inefficiencies in the existing in-memory scan implementation:
{code}
spark.range(N)
  .selectExpr("id", "floor(rand() * 10000) as k")
  .createOrReplaceTempView("test")
val ds = spark.sql("select count(k), count(id) from test").cache()
ds.collect()
ds.collect()
{code}

There are many reasons why caching is slow. The biggest is that compression takes a long time. The second is that there are a lot of virtual function calls in this hot code path since the rows are processed using iterators. Further, the rows are converted to and from ByteBuffers, which are slow to read in general.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org