[GitHub] [arrow-datafusion] alamb opened a new issue #106: [Datafusion] Optimize literal expression evaluation

[GitBox <gi...@apache.org>]

alamb opened a new issue #106:
URL: https://github.com/apache/arrow-datafusion/issues/106


   *Note*: migrated from original JIRA: https://issues.apache.org/jira/browse/ARROW-10243
   
   While benchmarking the tpch query I noticed that the physical literal expression takes up a sizable amount of time. I think the creation of the corresponding array for numeric literals can be speed up by creating Buffer and ArrayData directly without going through a builder. That also allows to skip building a null bitmap for non-null literals.
   
   I'm also thinking whether it might be possible to cache the created array. For queries without a WHERE clause, I'd expect all batches except the last to have the same length. I'm not sure though where to store the cached value.
   
   Another possible optimization could be to cast literals already on the logical plan side. In the tpch query the literal `1` is of type `u64` in the logical plan and then needs to be processed by a cast kernel to convert to `f64` for usage in an arithmetic expression.
   
   The attached flamegraph is of 10 runs of tpch, with the data being loaded into memory before running the queries (See ARROW-10240).
   
   {code}
   flamegraph ./target/release/tpch --iterations 10 --path ../tpch-dbgen --format tbl --query 1 --batch-size 4096 -c1 --load
   {code}


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-datafusion] alamb commented on issue #106: [Datafusion] Optimize literal expression evaluation

[GitBox <gi...@apache.org>]

alamb commented on issue #106:
URL: https://github.com/apache/arrow-datafusion/issues/106#issuecomment-826830937


   Comment from Jorge Leitão(jorgecarleitao) @ 2020-10-09T15:38:07.596+0000:
   <pre>All great ideas. Yes!</pre>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org