[I] ci: shard the Iceberg extensions test task [datafusion-comet]

["erikbogado-nstech (via GitHub)" <gi...@apache.org>]

erikbogado-nstech opened a new issue, #6101:
URL: https://github.com/apache/datafusion-comet/issues/6101

   ## Problem
   
   The Iceberg core tests are sharded by a task-agnostic class-name predicate, but `iceberg-spark-extensions` runs unsharded (`iceberg_spark_test_reusable.yml:221-227`) and measured **55m24s** in queue run `35022044196` — the single longest job in that run and the largest ownable saving.
   
   ## Change
   
   - Give the extensions job the same shard matrix: pass `-PcometShardIndex` / `-PcometShardCount` and `-PcometShardTask` for the extensions task, with a task-specific artifact name.
   - Add an extensions coverage check alongside the existing core one.
   
   ## Expected gain
   
   55m24s unsharded → ~15-20 min ⇒ ~35 queue-min/run × ~38 runs ≈ **1,300 runner-min/72 h (~2 %)**. The largest single ownable saving.
   
   ## Constraint
   
   `dev/ci/check-iceberg-shards.py:43` hard-codes `SHARD_COUNT = 4` and asserts every manifest's count equals it, so extensions must use count 4 (or the constant must become per-task). The Gradle predicate is a pure hash of the class name and is task-agnostic (`dev/ci/iceberg-test-shards.gradle`).
   
   ## Verification
   
   - One label run produces balanced per-shard candidate counts (extensions balance is unverified until then) and the coverage check passes: the union of shard candidates equals the unsharded inventory.
   - Queue wall clock for the extensions job 55m24s → target ~15-20 min.
   
   ## Risk / rollback
   
   Shard imbalance could make one shard as slow as the unsharded job; the coverage check prevents dropped tests. Revert the workflow and the matrix emitter.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [I] ci: shard the Iceberg extensions test task [datafusion-comet]

["erikbogado-nstech (via GitHub)" <gi...@apache.org>]

erikbogado-nstech closed issue #6101: ci: shard the Iceberg extensions test task
URL: https://github.com/apache/datafusion-comet/issues/6101


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org