[jira] [Created] (DRILL-5957) Wire protocol versioning, version negotiation

["Paul Rogers (JIRA)" <ji...@apache.org>]

Paul Rogers created DRILL-5957:
----------------------------------

             Summary: Wire protocol versioning, version negotiation
                 Key: DRILL-5957
                 URL: https://issues.apache.org/jira/browse/DRILL-5957
             Project: Apache Drill
          Issue Type: Improvement
    Affects Versions: 1.11.0
            Reporter: Paul Rogers


Drill has very limited support for evolving its wire protocol. As Drill becomes more widely deployed, this limitation will constrain the project's ability to rapidly evolve the wire protocol based on user experience to improve simplicitly, performance or minimize resource use.

Proposed is a standard mechanism to version the API and negotiate the API version between client and server at connect time. The focus here is between Drill clients (JDBC, ODBC) and the Drill server. The same mechanism can also be used between servers to support rolling upgrades.

This proposal is an outline; it is not a detailed design. The purpose here is to drive understanding of the problem. Once we have that, we can focus on the implementation details.

h4. Problem Statement

The problem we wish to address here concerns both the _syntax_ and _semantics_ of API messages. Syntax concerns:

* The set of messages and their sequence
* The format of bytes on the wire
* The format of message packets

Semantics concerns:

* The meaning of each field.
* The layout of non-message data (vectors, in Drill.)

We wish to introduce a system whereby both syntax and semantics can be evolved in a controlled, known manner such that:

* A client of version x can connect to, and interoperate with, a server in a range of versions (x-y, x+z) for some values of y and z.

For example, version x of the Drill client is deployed in the field. It must connect to the oldest Drill cluster available to that client. (That is it must connect to servers up to y versions old.) During an upgrade, the server may be upgraded before the client. Thus, the client must also work with servers up to z versions newer than the client.

If we wish to tackle rolling upgrades, then y and z can both be 1 for server-to-server APIs. A version x server will talk with (x-1) servers when the cluster upgrades to x, and will talk to (x+1) servers when the cluster is upgraded to version (x+1).

h4. Current State

Drill currently provides some ad-hoc version compatibility:

* Slow change. Drill's APIs have not changed much since Drill 1.0, thereby avoiding the issue.
* Protobuf support. Drill uses Protobuf for message bodies, leveraging that format's ability to absorb the additional or deprecation of individual fields.
* API version number. The API holds a version number, though the code to use it is rather ad-hoc.

The above has allowed clever coding to handle some version changes, but each is a one-off, ad-hoc collision. The recent security work is an example that, with enough effort, ad-hoc solutions can be found.

The above cannot handle:

* Change in the message order
* Change in the "pbody/dbody" structure of each message.
* Change in the structure of serialized value vectors.

As a result, the current structure prevents any change to Drill's core mechanism, value vectors, as there is no way or clients and servers to negotiate the vector wire format. For example, Drill cannot adopt Arrow because a pre-Arrow client would not understand "dbody" message parts encoded in Arrow format and visa-versa.

h4. API Version

The core of the proposal is to introduce an API version. This is a simple integer which is incremented each time that a breaking change is made to the API. (If the change can be absorbed by the Protobuf mechanism, then it is not a breaking change.) Note that the API version *is not* the same as the product version. Two different Drill versions may have the same API version if nothing changed in the API.

h4. Version Negotiation

Given a set of well-defined protocol versions, we can next define the version negotiation protocol between client and server:

* The client connects and sends a "hello" message that identifies the range of API versions that it supports, with the newest version being the version of the client itself.
* The server receives the message and computes the version of the session as the newest client version the the server supports.
* The server returns this version to the client which switches to the selected API version. (The server returns an error, and disconnects, if there is no common version.)
* The server and client use only messages valid for the given API version. This may mean converting data from one representation to another.

The above is pretty standard.

h4. Backward Compatibility Implementation

Consider a server that must work with its own version (version c) and, say, two older versions (a and b).

In most cases, changes across versions are minor. Perhaps version b introduced a better error reporting format (akin to SQLWARN and SQLERROR codes). Version c may have change the layout of a particular vector (or introduce a new vector type.) How does the server handle these when talking with older clients?

If a version c server talks to a version a client, then it must retain the old version a way of reporting errors. If the same version talks to a version b or c server, it uses the new form. This simply means that, at the point that an error response is sent to the client, the server selects either the a version of that message or the b version.

Since the a version previously existed, and the b version was added, it is straightforward (if tedious) or the developer to retain both versions and use them depending on the client version.

Now, consider version c that introduced a new vector format. Here, the server must transcode the data to the a version for clients of version a or b. Again, since the a version existed, and the c version was added, the developer is aware of both. The transcoding step is new, and does introduce a performance hit. But, it does allow the newer format to be rewritten in the older wire format.

For example, perhaps version c modified offset vectors to encode the end position of each row, rather than the start position (with the start position of the first row implied to be at 0.) This causes the offset vector to shrink by one position. Because of power-of-two rounding, this may halve the memory required. If the server talks to a version a or b client, then it simply transcodes the vector by shifting it upward and inserting the required 0 value at the zero position.

The same logic applies to the client as well, when receiving data. A version c client, when working with a version a server, must transcode the old offset vector format to the c version.

h4. Bootstrap

The above is fine for new Drill versions. But, how would we get started? This is the bootstrap problem: we can't change to the new version until both clients and servers understand the new version. But, since Drill does not currently support versioning, we can't introduce versioning until Drill understands versioning: a catch-22 situation.

We can observe that, initially, we are mostly concerned with client/server communication: old clients will exist in the field even after the upgrade to the version-aware server. We also observe that the client initiates the conversation.

One solution is to leverage the existing client "hello" message. Suppose we introduce the above versioning protocol in API version 10 (say). When a 9 or older version client connects, it will send the existing Protobuf "hello" message. The version 10 server can use that to detect the older client, and immediately switch to the old (unversioned) protocol.

Version 10 clients introduce a new V2 "hello" message that contains the schema negotiation information. If the server receives this, then the server knows it is working with a version 10 (or later) client, and will use that protocol instead, complete with version negotiation and adaptation as described above.

h4. Alternatives

Suppose that we decide not to introduce versioning at the API level, but instead soldier on with what we have. We will be able to:

* Carefully introduce new messages and new Protobuf fields.

We will *not* be able to:

* Change the wire format for any non-Protobuf data, especially value vectors.

The result is that we will be frozen at the Drill 1.0 version of the value vector format (and so we must hope that that original format was prescient about our future needs.) We cannot improve vector efficiency, nor can we switch to the Arrow format.

Unfortunately, our competitors will continue to move forward with their formats, placing Drill at a competitive disadvantage. So, in order to remain competitive, we have little alternative.
 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)