Format plugin configs should be immutable

[Paul Rogers <pa...@yahoo.com.INVALID>]

Hi All,

This is a quick note for any of you who create or work on format plugins in Drill. You will see that all existing plugins have been modified so that config properties are immutable. This note will explain why.

Drill uses storage and format plugins as keys into an internal map. (That's right: the whole plugin config is the key, not just the name.) As you might expect, things get out of sync if we change the fields used to compute the key hash. So, all storage and format plugin config fields must be immutable.

As it turns out, Drill has a very helpful feature: table functions which allow you to set format plugin properties within your SQL query. That code has historically required that all your format plugin fields be both public and mutable. (The implementation wrote directly to these public fields.)

As you can see, that created a contradiction: internal maps demand immutable fields, table functions want mutable fields. DRILL-6168 solves this by using a different way (based on JSON serialization) to implement table functions. (That change allow allows table functions to inherit properties from the base config stored in ZK.) This fix allowed us to change all existing format plugin configs to have immutable properties.


What this means for you is that, in your new code, please follow the patterns in the latest master: please make storage and plugin fields immutable.

Thanks,
- Paul