You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@celeborn.apache.org by "laxmanmamididd (via GitHub)" <gi...@apache.org> on 2026/09/01 17:00:35 UTC
[I] IndexOutOfBoundsException(-1) in StoragePolicy.createFileWriter: revived PartitionLocation assigned storage type HDD though never configured [celeborn]
laxmanmamididd opened a new issue, #3836:
URL: https://github.com/apache/celeborn/issues/3836
## Environment
- Celeborn 0.6.3 (server), Spark 4.0.1 client (`celeborn-client-spark-4-shaded_2.13-0.6.3`)
- Worker config: `celeborn.storage.availableTypes=SSD,S3`, `celeborn.worker.storage.storagePolicy.createFilePolicy=SSD,S3`, `celeborn.worker.storage.storagePolicy.evictPolicy=SSD,S3`
- Only one local disk configured per worker (`disktype=SSD`), no HDD directories configured anywhere.
- 4 workers, `celeborn.client.shuffle.replication.factor=2`
## What's happening
Under a real ~200GB shuffle load (`org.apache.spark.examples.GroupByTest`), every worker eventually throws:
```
ERROR StorageManager: Create partition data writer failed
java.lang.IndexOutOfBoundsException: -1
at scala.collection.LinearSeqOptimized.apply(LinearSeqOptimized.scala:67)
at scala.collection.immutable.List.apply(List.scala:91)
at org.apache.celeborn.service.deploy.worker.storage.StoragePolicy.$anonfun$createFileWriter$6(StoragePolicy.scala:197)
at org.apache.celeborn.service.deploy.worker.storage.StoragePolicy.createFileWriter(StoragePolicy.scala:196)
at org.apache.celeborn.service.deploy.worker.storage.StoragePolicy.createFileWriter(StoragePolicy.scala:60)
at org.apache.celeborn.service.deploy.worker.storage.PartitionDataWriter.<init>(PartitionDataWriter.java:91)
at org.apache.celeborn.service.deploy.worker.storage.StorageManager.createPartitionDataWriter(StorageManager.scala:477)
at org.apache.celeborn.service.deploy.worker.Controller.createPartitionDataWriter(Controller.scala:344)
at org.apache.celeborn.service.deploy.worker.Controller.createWriter$1(Controller.scala:290)
at org.apache.celeborn.service.deploy.worker.Controller.$anonfun$createWriters$1(Controller.scala:293)
...
at org.apache.celeborn.service.deploy.worker.Controller.org$apache$celeborn$service$deploy$worker$Controller$$handleReserveSlots(Controller.scala:210)
```
The full log line for the failing `PartitionLocation` shows:
```
Controller: Create FileWriter for PartitionLocation[
id-epoch:5-1
host-rpcPort-pushPort-fetchPort-replicatePort:[...]
mode:PRIMARY
...
storage hint:StorageInfo{type=HDD, mountPoint='', finalResult=false, filePath=, fileSize=0, chunkOffsets=[]}
...] celeborn-data03-coldtier-fill5-<id>-0-0 failed.
```
Note the `id-epoch:5-1` — this is epoch 1, i.e. a **revived** partition location (not the initial epoch-0 allocation), and its `storage hint` reports `type=HDD`.
## Why this crashes
`StoragePolicy.createFileWriter` (the non-`evict` overload, called via `PartitionDataWriter`'s constructor → `StorageManager.createPartitionDataWriter`) computes:
```scala
val tryCreateFileTypeIndex = order.get.indexOf(
partitionDataWriterContext.getPartitionLocation.getStorageInfo.getType.name())
```
Since `order` is our configured create-file order (`["SSD", "S3"]`) and the assigned type is `"HDD"`, `indexOf` returns `-1`, and the subsequent `order.get(tryCreateFileTypeIndex)` (`StoragePolicy.scala:197`) throws `IndexOutOfBoundsException: -1`.
## The actual question
**Why does a revived `PartitionLocation` get assigned storage type `HDD`** when nothing in the cluster's config uses HDD anywhere? `StorageInfo`'s own no-arg constructor defaults `type` to `MEMORY`, not `HDD`, so this isn't a generic uninitialized-field default — something in the revive/reallocation path (looks like it should be in `SlotsAllocator.getStorageInfo` or the master's revive handler, but I wasn't able to pin down the exact assignment) appears to be defaulting to `HDD` specifically for revived locations, independent of what disk types the target worker(s) actually have configured.
This reproduces reliably (100% of large-shuffle runs so far) once enough map/reduce tasks trigger at least one push-retry/revive cycle. Small jobs (a handful of reducers, little data) never hit it.
Happy to provide more repro detail, full worker/master logs, or test against a patch if pointed at the right code path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] IndexOutOfBoundsException(-1) in StoragePolicy.createFileWriter: revived PartitionLocation assigned storage type HDD though never configured [celeborn]
Posted by "laxmanmamididd (via GitHub)" <gi...@apache.org>.
laxmanmamididd closed issue #3836: IndexOutOfBoundsException(-1) in StoragePolicy.createFileWriter: revived PartitionLocation assigned storage type HDD though never configured
URL: https://github.com/apache/celeborn/issues/3836
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] IndexOutOfBoundsException(-1) in StoragePolicy.createFileWriter: revived PartitionLocation assigned storage type HDD though never configured [celeborn]
Posted by "laxmanmamididd (via GitHub)" <gi...@apache.org>.
laxmanmamididd commented on issue #3836:
URL: https://github.com/apache/celeborn/issues/3836#issuecomment-5497791487
Found the actual root cause on our end — this isn't a Celeborn bug, it's a config gap in our own deployment. `celeborn.storage.availableTypes` defaults to `HDD` (per `CelebornConf.ACTIVE_STORAGE_TYPES`, category applies to master/worker/**client**) if left unset. Our worker/master config correctly set it to `SSD,S3`, but we never set the matching client-side Spark conf (`spark.celeborn.storage.availableTypes`) — so the client fell back to Celeborn's `HDD` default, and that mismatch is what produced a revived `PartitionLocation` hinted `HDD` that our worker's `SSD,S3` create-file-order list doesn't contain. Setting `spark.celeborn.storage.availableTypes=SSD,S3` on the client resolved it — confirmed with a full ~200GB shuffle job completing successfully end to end, where every previous attempt (without the fix) crashed the same way. Sorry for the noise, and thanks for Celeborn!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org