You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "wForget (via GitHub)" <gi...@apache.org> on 2026/07/22 04:16:05 UTC
[I] [VL][UNIFFLE] Fix incorrect task attempt ID used for Uniffle block IDs [gluten]
wForget opened a new issue, #12594:
URL: https://github.com/apache/gluten/issues/12594
### Backend
VL (Velox)
### Bug description
### Problem
In the columnar shuffle path, [`UniffleShuffleManager#getWriter`](https://github.com/apache/gluten/blob/main/backends-velox/src-uniffle/main/java/org/apache/spark/shuffle/gluten/uniffle/UniffleShuffleManager.java) passes `context.taskAttemptId()` directly to `VeloxUniffleColumnarShuffleWriter` as the task attempt ID used for generating Uniffle block IDs.
Spark's `TaskContext.taskAttemptId()` is a globally monotonically increasing value. It may exceed the number of bits reserved for the task attempt ID in Uniffle's `BlockIdLayout`, causing columnar shuffle writes to fail in applications with sufficiently large task attempt IDs.
Uniffle's standard `RssShuffleManager` avoids this by deriving a stage-scoped ID:
```java
getTaskAttemptIdForBlockId(context.partitionId(), context.attemptNumber())
```
The row-based shuffle path already uses the standard Uniffle implementation and is not affected.
### Proposed fix
Compute the block ID task attempt ID using the inherited helper and pass it to `VeloxUniffleColumnarShuffleWriter`:
```java
long rssTaskAttemptId =
getTaskAttemptIdForBlockId(context.partitionId(), context.attemptNumber());
```
Replace the current constructor argument:
```java
context.taskAttemptId()
```
with:
```java
rssTaskAttemptId
```
The existing string-form `taskId` can remain unchanged because it is not encoded into the Uniffle block ID.
### Expected behavior
The Gluten Uniffle columnar shuffle writer should use the same task attempt ID encoding as Uniffle's standard `RssShuffleManager`, preventing block ID overflow for large Spark task attempt IDs.
### Gluten version
_No response_
### Spark version
None
### Spark configurations
_No response_
### System information
_No response_
### Relevant logs
```bash
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL][UNIFFLE] Fix incorrect task attempt ID used for Uniffle block IDs [gluten]
Posted by "wForget (via GitHub)" <gi...@apache.org>.
wForget closed issue #12594: [VL][UNIFFLE] Fix incorrect task attempt ID used for Uniffle block IDs
URL: https://github.com/apache/gluten/issues/12594
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org