Re: [I] Hudi configs cannot take effect during or after the table has been created [hudi]

["rangareddy (via GitHub)" <gi...@apache.org>]

rangareddy commented on issue #17328:
URL: https://github.com/apache/hudi/issues/17328#issuecomment-5351232106

   This issue was reviewed as part of the JIRA-migrated backlog triage (HUDI-8665).
   
   **Findings: needs a `spark-sql` reproduction on current `master` before it can be dispositioned.**
   
   The report is specific and actionable as written - `SET hoodie.metadata.enable`, `hoodie.metadata.index.column.stats.enable` and `hoodie.enable.data.skipping` only take effect when issued before `CREATE TABLE`, and are ignored inside the `OPTIONS` clause. What it needs is confirmation against a current build, because the config-resolution path has changed substantially since it was filed.
   
   I did not want to guess at this one from code alone: whether a `SET` loses to a table property, an `OPTIONS` entry, or an inferred default depends on the precedence order that `HoodieCatalog` / `HoodieWriterUtils` apply at create time, and that is not decidable by reading without running it.
   
   One pointer that may be the same root cause, since it is independently confirmed: #17398 (HUDI-9153) documents that `HoodieConfig.setDefaults` iterates `Class#getDeclaredFields()` and writes each resolved default into `props` as it goes, so a `ConfigProperty` infer function calling `contains(...)` cannot tell "the user set this" from "we just defaulted it". That is exactly the failure shape described here. Worth checking whether these two are the same bug before doing independent work on them.
   
   Keeping this open; a repro on a recent version (ideally without the 1TB dataset) would unblock it.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org