Next set of technical things to work on?

[Jignesh Patel <jm...@gmail.com>]

Hello folks, 

 

Would love to have a discussion of what are the next set of technical things to work on. To begin, here is a summary of what has gone into the PRs recently. 

 

#1: A large amount of focus has been on the query execution and optimization methods. The key PRs here are PR #113, Issue #69/PR172. These added, as noted in the PR discussion, an aggressive semi-join technique. A paper describing this work will appear in VLDB 2017 at the end of the summer. The paper is up at http://www.vldb.org/pvldb/vol10/p889-zhu.pdf. There are continual changes to the optimizer that largely Jianqiao fixes. AFAIK, there are no major features that are currently planned on this component. Please chime in if this is not true. 

 

#2: There is work on adding support for union/intersection operator lead by our newest committer, Tianrun Li. This originated  from him putting GRAIL on top of Quickstep. GRAIL is a way to map BSP-style graph analytics on a relational core, and was published (as a separate project) two years ago. See http://cidrdb.org/cidr2015/Papers/CIDR15_Paper20.pdf 

 

#3: The big active feature right now is on the distributed version, lead by Zuyu. (Zuyu please step in to correct the notes below as necessary.) In various discussions and PRs the plan he has put forward is to: 
Work on the partitioning in Quickstep as that is critical in the distributed version. There was robust discussion on this around Issue 73/PR 217
Moving data around nodes. The idea here is to use the design that we have with blocks having unique id, and letting a node pull a block from a remote node when it can’t find a block. This would use the GPRC mechanism that we have. I think it would be good to have a discussion of this design and get input from everyone who can help. This is clearly a big feature. Zuyu: Do you want to summarize your design in more detail when you have worked it out?
 

#4: Most of the Wisconsin folks are off for the summer, so I think things are going to slow down a bit on the development pace.  

 

As a note, we have been slack in having discussions of design on the dev list. I apologize for my part on this front. Having a discussion on the dev list is a great idea, and healthy for all in the long-run. So please feel free to chime in with your views and feel free to make a design proposal for specific parts that you think need work on.  

 

Happy Memorial Day everyone! 

 

Cheers,

Jignesh