Re: [I] If schema provider is specified then use it for Row Source [hudi]

["nsivabalan (via GitHub)" <gi...@apache.org>]

nsivabalan commented on issue #15466:
URL: https://github.com/apache/hudi/issues/15466#issuecomment-4795430472

   This appears to be addressed on current master. `Source#fetchNext` (the `final` wrapper around `readFromCheckpoint`) replaces the InputBatch's schema provider with the user-specified one when present:
   
   ```java
   // hudi-utilities/.../sources/Source.java
   public final InputBatch<T> fetchNext(Option<Checkpoint> lastCheckpoint, long sourceLimit) {
     ...
     // If overriddenSchemaProvider is passed in CLI, use it
     return overriddenSchemaProvider == null ? batch
         : new InputBatch<>(batch.getBatch(), batch.getCheckpointForNextBatch(), overriddenSchemaProvider);
   }
   ```
   
   The `overriddenSchemaProvider` is populated from `StreamContext#getSchemaProvider()` in the `Source` constructor, so any user-specified `SchemaProvider` (with its default values) overrides the `RowBasedSchemaProvider` that `RowSource#readFromCheckpoint` builds from the dataset.
   
   Additionally, the Row→Avro conversion in `SourceFormatAdapter#fetchNewDataInAvroFormat` was tightened in PR #17946 (commit 1ff0506) to use the original `FileBased`/`SchemaRegistry` provider's source schema when applicable (via `UtilHelpers#getOriginalSchemaProvider`).
   
   If you can still reproduce default values being dropped on a recent master with a user-specified schema provider, please share the config — otherwise this can likely be closed.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org