You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "wForget (via GitHub)" <gi...@apache.org> on 2026/07/22 04:16:05 UTC

[I] [VL][UNIFFLE] Fix incorrect task attempt ID used for Uniffle block IDs [gluten]

wForget opened a new issue, #12594:
URL: https://github.com/apache/gluten/issues/12594

   ### Backend
   
   VL (Velox)
   
   ### Bug description
   
   ### Problem
   
   In the columnar shuffle path, [`UniffleShuffleManager#getWriter`](https://github.com/apache/gluten/blob/main/backends-velox/src-uniffle/main/java/org/apache/spark/shuffle/gluten/uniffle/UniffleShuffleManager.java) passes `context.taskAttemptId()` directly to `VeloxUniffleColumnarShuffleWriter` as the task attempt ID used for generating Uniffle block IDs.
   
   Spark's `TaskContext.taskAttemptId()` is a globally monotonically increasing value. It may exceed the number of bits reserved for the task attempt ID in Uniffle's `BlockIdLayout`, causing columnar shuffle writes to fail in applications with sufficiently large task attempt IDs.
   
   Uniffle's standard `RssShuffleManager` avoids this by deriving a stage-scoped ID:
   
   ```java
   getTaskAttemptIdForBlockId(context.partitionId(), context.attemptNumber())
   ```
   
   The row-based shuffle path already uses the standard Uniffle implementation and is not affected.
   
   ### Proposed fix
   
   Compute the block ID task attempt ID using the inherited helper and pass it to `VeloxUniffleColumnarShuffleWriter`:
   
   ```java
   long rssTaskAttemptId =
       getTaskAttemptIdForBlockId(context.partitionId(), context.attemptNumber());
   ```
   
   Replace the current constructor argument:
   
   ```java
   context.taskAttemptId()
   ```
   
   with:
   
   ```java
   rssTaskAttemptId
   ```
   
   The existing string-form `taskId` can remain unchanged because it is not encoded into the Uniffle block ID.
   
   ### Expected behavior
   
   The Gluten Uniffle columnar shuffle writer should use the same task attempt ID encoding as Uniffle's standard `RssShuffleManager`, preventing block ID overflow for large Spark task attempt IDs.
   
   ### Gluten version
   
   _No response_
   
   ### Spark version
   
   None
   
   ### Spark configurations
   
   _No response_
   
   ### System information
   
   _No response_
   
   ### Relevant logs
   
   ```bash
   
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [I] [VL][UNIFFLE] Fix incorrect task attempt ID used for Uniffle block IDs [gluten]

Posted by "wForget (via GitHub)" <gi...@apache.org>.
wForget closed issue #12594: [VL][UNIFFLE] Fix incorrect task attempt ID used for Uniffle block IDs
URL: https://github.com/apache/gluten/issues/12594


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org