[GitHub] [drill] jnturton edited a comment on issue #2421: ValueVectors replacement

[GitBox <gi...@apache.org>]

jnturton edited a comment on issue #2421:
URL: https://github.com/apache/drill/issues/2421#issuecomment-1005676173


   A few, informational notes from me.  First, I wondered how it could be that benchmark-busting Impala manages to load SIMD registers from a row oriented memory layout then found the following [in here](http://cidrdb.org/cidr2015/Papers/CIDR15_Paper28.pdf).
   
   > We are also considering switching to a columnar canonical in-memory format for data that needs to be materialized during query processing, in order to take advantage of SIMD instructions 
   
   I wonder whether they switched?  I suppose I need only go over to apache/impala to find out.  EDIT: [I don't think they did](https://git-wip-us.apache.org/repos/asf?p=impala.git;a=blob;f=be/src/runtime/row-batch.cc;h=aed2e116781718e715e5ce01db996412eef0aa9f;hb=HEAD).  I can't speak for anyone else but some of my important Drill workloads actually have been bulk floating point arithmetic (statistics, probabilities) where you do get a major boost from SIMD.  That said, I probably did even more work doing filtering, sorting and aggregating the same data in Drill before it was ready for arithmetic.
   
   Second, if you did want to do SIMD properly in Java (i.e. not praying for auto-vectorisation) then are you looking at JEP 338 and do [you have some vector types from JDK dictated to you](https://openjdk.java.net/jeps/338)?  Would these vectors work with our direct memory and Netty-based code?
   
   Third, would our direct memory code remain unchanged if we replace ValueVectors or would we want to start accessing direct memory through [JEP 393](https://openjdk.java.net/jeps/393).  Or... are [the high tech garbage collectors in recent JVMs](https://blogs.oracle.com/javamagazine/post/understanding-the-jdks-new-superfast-garbage-collectors) so good that we would laugh off the burden direct memory and its management entirely?
   
   Re. SIMD, I know there are positions like "Forget SIMD" that have been stated, but I wanted to note this stuff down for us anyway.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@drill.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org