Re: [PR] Spark: Add scd_type2 option to create_changelog_view procedure [iceberg]

["moomindani (via GitHub)" <gi...@apache.org>]

moomindani commented on PR #17765:
URL: https://github.com/apache/iceberg/pull/17765#issuecomment-5591058614

   @szehon-ho a nudge on this one — the question at the end of my 26 August comment is the only thing holding it, and I would rather have an answer either way than leave it sitting.
   
   To make it cheap to answer: the three routes to a long-lived SCD Type-2 table were (1) dual write, (2) SCD Type-2 as the base table, (3) derive from the changelog and store it. This PR is only the derivation step of (3) — it stores nothing itself. If you meant (1) or (2), those are different features and I do not claim this covers them.
   
   One correction: that comment pointed at `ef48e01a1` for the documentation, but I rebased the branch on 30 August and that SHA no longer exists. The docs are now in `40f83f48c`, under "Building a Long-Lived SCD Type-2 Table" in `docs/docs/spark-procedures.md`; it states the retention bound plainly and shows the incremental loop.
   
   And if your conclusion is that the retention window makes this not worth carrying inside the procedure, please say so and I will close the PR. Rebased onto current `main`, CI green (38/38).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org