[I] [Feature] [spark] Support failure-safe staged REPLACE TABLE AS SELECT [paimon]

["Akash3121 (via GitHub)" <gi...@apache.org>]

Akash3121 opened a new issue, #9994:
URL: https://github.com/apache/paimon/issues/9994

   ### Search before asking
   
   - [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.
   
   
   ### Motivation
   
    Spark can route compatible REPLACE TABLE AS SELECT operations through its staged/atomic execution path, but Paimon's current stageReplace implementation replaces the live table before the query output is written.
    
    SparkCatalog#stageReplaceInternal currently calls catalog.replaceTable immediately and returns a RollbackStagedTable with an empty abort action. The underlying replace implementation is documented as non-atomic and can truncate the table before appending the new schema.
    
    For example:
    ```sql
    SET spark.sql.ansi.enabled=true;
    
    CREATE TABLE src(s STRING) USING paimon;
    INSERT INTO src VALUES ('bad');
    
    CREATE TABLE target(id BIGINT, data STRING) USING paimon;
    INSERT INTO target VALUES (1, 'old');
    
    CREATE OR REPLACE TABLE target USING paimon
    AS SELECT CAST(s AS INT) AS id FROM src;
    ```
    The cast fails during the write. However, the target may already have been truncated and changed to the replacement schema.
    
    A failed staged RTAS should leave the original target schema, snapshots, options, and data unchanged. Losing the live table after a failed replacement is especially surprising when Spark selected an execution node named AtomicReplaceTableAsSelectExec.
   
   
   ### Solution
   
   Stage replacement output separately and publish it only from commitStagedChanges.
    
    Possible approaches include:
    
    1. Write into an isolated temporary branch, table, or location.
    2. Keep the existing target untouched during the query write.
    3. Atomically or transactionally publish the staged replacement from commitStagedChanges where the catalog supports it.
    4. Delete staged artifacts from abortStagedChanges.
    5. If a catalog cannot provide safe staging, do not advertise/use Spark's atomic replacement path and fail clearly instead of exposing a false rollback contract.
    
    Suggested tests should inject an executor-side failure after staging and verify that the original schema, latest snapshot, rows, and options are unchanged. Coverage should include Spark 3 and Spark 4 and the filesystem, Hive, and REST catalog implementations.
   
   ### Anything else?
   
    Relevant code:
    
    - PaimonReplaceTableAsSelectStrategy selects Spark's atomic RTAS execution path.
    - SparkCatalog#stageReplaceInternal eagerly invokes catalog.replaceTable.
    - The returned RollbackStagedTable receives an empty abort action.
    - AbstractCatalog documents replace as non-atomic on failure.
    
    This is a larger feature than the other candidates and likely needs maintainer agreement on the staging model before implementation.
   
   ### Are you willing to submit a PR?
   
   - [x] I'm willing to submit a PR!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Feature] [spark] Support failure-safe staged REPLACE TABLE AS SELECT [paimon]

["Akash3121 (via GitHub)" <gi...@apache.org>]

Akash3121 commented on issue #9994:
URL: https://github.com/apache/paimon/issues/9994#issuecomment-5747711468

   I'm willing to submit a pr for this issue


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Feature] [spark] Support failure-safe staged REPLACE TABLE AS SELECT [paimon]

["LuciferYang (via GitHub)" <gi...@apache.org>]

LuciferYang commented on issue #9994:
URL: https://github.com/apache/paimon/issues/9994#issuecomment-6435608500

   @Akash3121 are you still working on this? I'm asking because of #9690 (Spark WAP). One way to make staged RTAS atomic reuses the same pieces: write the new schema and data to a hidden branch, fast-forward main in `commitStagedChanges`, and drop the branch in `abortStagedChanges`. That needs `fast_forward` to be atomic first, and today it isn't (see my comment on #9690). If you have a different approach in mind, I'd like to hear it before either of us writes code.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org