[jira] [Created] (HDDS-5129) Support new field in datanode container descriptors

["Marton Elek (Jira)" <ji...@apache.org>]

Marton Elek created HDDS-5129:
---------------------------------

             Summary: Support new field in datanode container descriptors
                 Key: HDDS-5129
                 URL: https://issues.apache.org/jira/browse/HDDS-5129
             Project: Apache Ozone
          Issue Type: Improvement
            Reporter: Marton Elek
            Assignee: Marton Elek


Each of the containers have a container descriptor in datanode which contains metadata (containerId, path to the metadata/chunks, schema version...).

These fields are checksummed in the same file one specific field should contain the checksum of the other (pre-defined) fields.

The checksum uses the pre-defined list of the properties to calculate the cheksum (KeyValueContainerData) 

{code}
  static {
    // Initialize YAML fields
    KV_YAML_FIELDS = Lists.newArrayList();
    KV_YAML_FIELDS.addAll(YAML_FIELDS);
    KV_YAML_FIELDS.add(METADATA_PATH);
    KV_YAML_FIELDS.add(CHUNKS_PATH);
    KV_YAML_FIELDS.add(CONTAINER_DB_TYPE);
    KV_YAML_FIELDS.add(SCHEMA_VERSION);
    KV_YAML_FIELDS.add(REPLICA_INDEX);
  }
{code}

This approach makes it impossible to extend the fields with a backward compatible way.

If a new field is added, the checksum will not match:

new code will calculate checksum = (old fields + new field), the old code will calculate checksum = (old fields)

Therefore, if the new code starts to create metadata (with checksum), old code will fail to read the descriptors because the checksum violation.

It would be better to use ALL the fields (except the checksum itself) for calculating the checksum. With this approach the old code can read the descriptor even if there are unknown fields included.

The only question: how can we switch to the new type of calculation.

1. Option1: Using upgrade framework

Using upgrade framework we can start to use any new typo of calculation after finalize step

2. Option2: Using new field

(Thanks to Uma, for the suggestion).

It is also possible to use a new field to store the new type of checkum. "checksum" field can be calculated as earlier, but we can calculate "fullchecksum" property based on all the properties.

This is a fully backward compatible way, but we should calculate both the checksum until the next major version where we can allow doing incompatible changes (possible with using upgrade framework).

Personally I would suggest to use option1 as we need an upgrade framework anyway, it may be better to avoid duplicated information.  





 



 




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org