[jira] [Created] (HUDI-8438) Fix table schema in commit metadata of table services

["Davis Zhang (Jira)" <ji...@apache.org>]

Davis Zhang created HUDI-8438:
---------------------------------

             Summary: Fix table schema in commit metadata of table services
                 Key: HUDI-8438
                 URL: https://issues.apache.org/jira/browse/HUDI-8438
             Project: Apache Hudi
          Issue Type: Improvement
          Components: multi-writer
            Reporter: Davis Zhang


related Jira https://issues.apache.org/jira/browse/HUDI-8219

 

In the Jira above we found issues with how table schema is resolved, where it can read a latest completed instant coming from table service and the schema in the commit metadata is a stale one.

The main reason is they don't go through org.apache.hudi.client.transaction.SimpleSchemaConflictResolutionStrategy#resolveConcurrentSchemaEvolution when it writes commit metadata to complete instant.

As a result, org.apache.hudi.common.table.TableSchemaResolver#getTableAvroSchemaFromSchemaEvolutionTimeline is used in org.apache.hudi.client.transaction.SimpleSchemaConflictResolutionStrategy#resolveConcurrentSchemaEvolution to skip instants from table services when fetching the table schema.

 

To fix the issue,

For clustering, the auto commit hard-coded to false, so at the time it tries to do commit, it goes a different commit code path at org.apache.hudi.client.HoodieFlinkTableServiceClient#completeClustering. Here we need to read the latest table schema and use that in commit metadata.

 

For compaction, in SimpleSchemaConflictResolutionStrategy#resolveConcurrentSchemaEvolution we should do the same thing of reading the latest table schema and return. We don't need to go with full schema evolution logic since compaction does not evolve schema so it must not hit any schema conflict.

 

For clean, rollback, archive their completed instant will be filtered out while reading the table schema, so it's fine if they do not come with one in their commit metadata.

 

After fixing these, we should have:

In the timeline, for all completed instant of COMMIT, DELTA_COMMNIT, REPLACEMENT_COMMIT, COMPACTION, the table schema "version" is monotoncially increasing, whoever comes later must use a more recent table schema instead of a stale one.

The getTableAvroSchemaFromSchemaEvolutionTimeline function call would be replace by normal getTableAvroSchema call since we don't need to filter out table service instant anymore 

 

 

 

What we need to change is after 
this.txnManager.beginTransaction(Option.of(clusteringInstant), Option.empty());



--
This message was sent by Atlassian Jira
(v8.20.10#820010)