Minutes from Hangout

[Charles Givre <cg...@gmail.com>]

Hello all, 
Here are my and Paul's notes from the Drill Hangout on Friday.

Charles Givre notes:
CI discussion: 
Open issue: Identify core tests:
1.  PR is OK

2.  Next Release:
	1.  Druid needs QA
	2.  Possible to release 1.18 in the next two months
	3.  Headers to Web UI
	4.  Specfic issue with column ordering
	5.  HTTP Plugin
	6.  Filter Pushdown Improvements?
	7.  Support for all format plugins for metastore

Start thinking about dates for next release. 


3.  Release Cadeence
	1.  (Vova). No critical regressions
	2.  Mongo Issues (DRILL-7628)
	3.  Runbook for issues that are not covered in unit tests

4. SPI Plugin Discussion: 
	1.  Goals:  Plugin easier to work with if it is in the Drill codebase


Paul's Notes:
CI Tools
- Faster: split tests, run subset
- How to test plugins that require external systems?
- Can we test only changed modules (especially for contrib modules?)

Next Release
- When? Advantage of three-month "train" model.
- Constraints: when CV can do it.
- Vova noted that pre-commit tests are run for every PR, any PMC can do other release steps
- Igor is working on a fix for project column order
- Vova is doing some work on format plugins
- Ensure release is stable. Last had 3 RCs due to bugs. Is costly to handle.
- Can we create a checklist of tasks for committers who check an RC?

SPI
- Outline of idea: easier to add third-party plugins
- Class path isolation
- Current plugin registry work is foundation
- Igor noted he has used the ServiceLoader classes and found them helpful
- How will we handle Protobuf issues (operator ID)?

Dynamic schema
- Drill's attraction is dynamic schema: infer from data
- Some data is messy, schema is ambiguous
- Drill obligated by SQL standard to provide consistent schema to xDBC clients
- Proposed solution: dynamic when data allows, else provide schema
- All storage plugins can optionally provide schema (or use Metastore)
- Once schema is available, deprecate schema change support (has never really worked)

Other topics
- Charles working on an XML reader. Challenges of decoding. How to handle deep nesting.  NOTE:  Development on XML reader is on hold at the moment with other projects.  I can share code if anyone is interested.
- Discussion of Drill as data integration tool
- Igor is starting to look at pulling the web server into module separate from exec