[jira] [Created] (HUDI-9597) Schema reconcilation issue

["Davis Zhang (Jira)" <ji...@apache.org>]

Davis Zhang created HUDI-9597:
---------------------------------

             Summary: Schema reconcilation issue
                 Key: HUDI-9597
                 URL: https://issues.apache.org/jira/browse/HUDI-9597
             Project: Apache Hudi
          Issue Type: Bug
            Reporter: Davis Zhang


on branch 0.14, we have 

 
{code:java}
// Save internal schema
private void saveInternalSchema(HoodieTable table, String instantTime, HoodieCommitMetadata metadata) {
  TableSchemaResolver schemaUtil = new TableSchemaResolver(table.getMetaClient());
  String historySchemaStr = schemaUtil.getTableHistorySchemaStrFromCommitMetadata().orElse("");
  FileBasedInternalSchemaStorageManager schemasManager = new FileBasedInternalSchemaStorageManager(table.getMetaClient());
  if (!historySchemaStr.isEmpty() || Boolean.parseBoolean(config.getString(HoodieCommonConfig.RECONCILE_SCHEMA.key()))) {
    InternalSchema internalSchema;
    Schema avroSchema = HoodieAvroUtils.createHoodieWriteSchema(config.getSchema(), config.allowOperationMetadataField());
    if (historySchemaStr.isEmpty()) {
      internalSchema = SerDeHelper.fromJson(config.getInternalSchema()).orElse(AvroInternalSchemaConverter.convert(avroSchema));
      internalSchema.setSchemaId(Long.parseLong(instantTime));
    } else {
      internalSchema = InternalSchemaUtils.searchSchema(Long.parseLong(instantTime),
          SerDeHelper.parseSchemas(historySchemaStr));
    }
    InternalSchema evolvedSchema = AvroSchemaEvolutionUtils.reconcileSchema(avroSchema, internalSchema); {code}
the AvroSchemaEvolutionUtils.reconcileSchema take 2 inputs

 

avroSchema - which is the writer schema populated in string format from the writer config

internalSchema - which is read from the branch

 

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)