[jira] [Created] (DRILL-7613) Revise, harden the preliminary storage plugin upgrade facility

["Paul Rogers (Jira)" <ji...@apache.org>]

Paul Rogers created DRILL-7613:
----------------------------------

             Summary: Revise, harden the preliminary storage plugin upgrade facility
                 Key: DRILL-7613
                 URL: https://issues.apache.org/jira/browse/DRILL-7613
             Project: Apache Drill
          Issue Type: Improvement
    Affects Versions: 1.17.0
            Reporter: Paul Rogers


Drill provides a way to upgrade storage plugins after installing a new Drill version. However the mechanism is very crude and error prone. It is based on overwriting the current contents of the persistent store with a new set of configs. However, doing so is likely to discard essential user config.

For example, we cannot upgrade format plugins individually. So, to add a new format plugin, we must overwrite, say, the {{dfs}} config. In so doing we may throw away the user's S3 or HDFS config.

Further, we don't want to reapply the upgrades on every restart. So, the mechanism has the ability to delete a file to mark the system as upgraded. There are several problems with this idea. First, any such file is likely to be in a jar as a resource, so is not deletable except in an IDE (when we would really *not* want to delete it.)

Suppose the user does an upgrade, suffers a ZK loss, and restores from backup. Drill will not know to re-upgrade ZK because the file is gone (assuming that the delete actually worked.) This shows that using a file to indicate ZK state is a poor implementation choice.

The code does not handle race conditions. If we bring up a cluster of 10 Drillbits, all 10 will race to perform upgrades.

The code has partial code to upgrade format plugins, but that code does not actually work as there is no no good way to do that upgrade. (Each DFS storage plugin has its own set of format plugins, unfortunately, and there is no code to find all such DFS storage plugins and apply format plugin updates.)

A better solution would be to:

* Store a version in the plugin registry. Upgrade only if the version in the registry is lower than the current Drillbit version.
* Better, provide a SQL command to force the upgrade. This allows users to do rolling v-1/v upgrades without the v-1 Drillbits seeing plugins that they cannot handle.
* Implement a race-condition-proof upgrade: select one Drillbit to do the upgrade and let the others wait. (Leader election.)
* Separate format plugins from the DFS storage plugin. Allow the same formats to be used across all configured DFS plugins. (There is no harm in offering a format for non-existent files.)
* Complete the work to upgrade the (now separate) format plugins so we can automatically roll out new formats without users having to do the upgrade manually.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)