Re: [I] Auto tune bulk insert parallelism #555 [hudi]

["nsivabalan (via GitHub)" <gi...@apache.org>]

nsivabalan commented on issue #14407:
URL: https://github.com/apache/hudi/issues/14407#issuecomment-4795108514

   Triage note: this long-standing request (HUDI-10, originally uber/hudi#555) appears to be addressed in current `master`.
   
   - `ParallelismHelper.deduceShuffleParallelism` (`hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/commit/ParallelismHelper.java`) returns the user-configured parallelism only when `> 0`; otherwise it falls back to the natural partition count of the incoming dataset.
   - `SparkBulkInsertHelper.bulkInsert` invokes this via `deduceShuffleParallelism(inputRecords, configuredParallelism)`, so when `hoodie.bulkinsert.shuffle.parallelism` is unset, bulk insert auto-tunes to the input parallelism instead of a static default.
   - This was introduced by [HUDI-5363](https://issues.apache.org/jira/browse/HUDI-5363) (commit `3a08bdc3f97`), which explicitly removed the previous static default of 200. Companion work: [HUDI-4924](https://issues.apache.org/jira/browse/HUDI-4924) auto-tunes dedup parallelism, and `SparkSampleWritesUtils` (HoodieStreamer) handles record-size–driven tuning.
   
   If the original ask was something more specific (e.g., size-aware tuning at the write-client level beyond the streamer's sample-writes path), please clarify; otherwise this can likely be closed.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org