[I] [tiering] State machine race: first tiering round fails with 'Scheduled to Tiered is not a valid state change' [fluss]

["qzyu999 (via GitHub)" <gi...@apache.org>]

qzyu999 opened a new issue, #3672:
URL: https://github.com/apache/fluss/issues/3672

   ## Summary
   
   The first tiering round for a newly created datalake-enabled table fails with a state machine validation error. Subsequent rounds succeed.
   
   ## Error
   
   `
   ERROR org.apache.fluss.server.coordinator.LakeTableTieringManager - Fail to change state for table 1 from Scheduled to Tiered as it's not a valid state change.
   ERROR org.apache.fluss.server.coordinator.LakeTableTieringManager - Fail to change state for table 1 from Scheduled to Scheduled as it's not a valid state change.
   `
   
   ## Expected Behavior
   
   The state machine should transition Scheduled  Pending  Tiering  Tiered without errors on the first tiering round.
   
   ## Actual Behavior
   
   The tiering job completes its work (S3 write succeeds, \storeLakeTableOffsetsFile()\ succeeds) but the Coordinator rejects the state transition because the table is still in \Scheduled\ state when the tiering result arrives. The expected intermediate transitions (Scheduled  Pending  Tiering) haven't occurred yet.
   
   The second tiering round works fine  by then the state machine has caught up.
   
   ## Steps to Reproduce
   
   1. Create a table with \'table.datalake.enabled' = 'true', 'table.datalake.freshness' = '60s'\
   2. Insert a small amount of data (e.g., 3 rows)
   3. Start the tiering service
   4. Observe Coordinator logs within the first freshness interval
   
   ## Environment
   
   - Fluss: built from main (post-PR #3550)
   - Flink: 1.20.1
   - Iceberg: 1.10.1 with Glue catalog
   - Infrastructure: ECS Fargate (single TabletServer, single bucket)
   
   ## Analysis
   
   Likely a timing issue: with a small dataset and fast S3 writes, the tiering job completes its first round before \LakeTableTieringManager\ has transitioned the table through the intermediate states. The state machine expects the sequence Scheduled  Pending  Tiering  Tiered, but the tiering completion arrives while still in Scheduled.
   
   This may be more likely with:
   - Small datasets (fast tiering)
   - Fast network to S3
   - Short freshness intervals
   
   ## Impact
   
   - First tiering round's snapshot is lost (not committed to Iceberg metadata)
   - Subsequent rounds work correctly
   - Data is eventually tiered, but there's a one-round delay on fresh tables


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [tiering] State machine race: first tiering round fails with 'Scheduled to Tiered is not a valid state change' [fluss]

["luoyuxia (via GitHub)" <gi...@apache.org>]

luoyuxia closed issue #3672: [tiering] State machine race: first tiering round fails with 'Scheduled to Tiered is not a valid state change'
URL: https://github.com/apache/fluss/issues/3672


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [tiering] State machine race: first tiering round fails with 'Scheduled to Tiered is not a valid state change' [fluss]

["luoyuxia (via GitHub)" <gi...@apache.org>]

luoyuxia commented on issue #3672:
URL: https://github.com/apache/fluss/issues/3672#issuecomment-5714133083

   Duplicate of #4406.
   
   Consolidating the reported state-transition errors in #4406, which includes a deterministic reproduction of duplicate completion reports and tracks the server-side idempotency fix, including preservation of tiering statistics.
   
   The exact first-round trigger described here has not been independently verified. The duplicate-completion reproduction does not establish snapshot loss; if this report persists after the fix, it should be investigated separately.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org