[PR] [flink] Fix range assignment with duplicate shuffle boundaries [paimon]

["liangjie3138 (via GitHub)" <gi...@apache.org>]

liangjie3138 opened a new pull request, #10318:
URL: https://github.com/apache/paimon/pull/10318

   ### Purpose
   
   Fix overlapping ranges in Flink `RangeShuffle` when sampling boundaries contain duplicate values. Assign non-boundary keys deterministically while preserving random distribution for equal keys to avoid data skew.
   
   ### Tests
   
   Added parameterized operator tests in `RangeShuffleTest` to verify that:
   
   - Keys below, between, and above repeated boundaries always enter the correct range.
   - Keys equal to repeated boundaries stay within the corresponding range group.
   - All-identical boundaries, distinct boundaries, and empty boundaries are handled correctly.
   - Each input record is emitted once with its key and payload preserved


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [flink] Fix range assignment with duplicate shuffle boundaries [paimon]

["JingsongLi (via GitHub)" <gi...@apache.org>]

JingsongLi commented on PR #10318:
URL: https://github.com/apache/paimon/pull/10318#issuecomment-5934470163

   Reviewed `627450ee76`: the change has end-to-end value for sorted output files, and I found no code issue in the boundary grouping, binary search, range partitioner or downstream sort path. I ran `RangeShuffleTest` locally with normal Maven checks: all 19 tests passed on both Flink 1/JDK 8 and Flink 2/JDK 17.
   
   One production-validation gap remains: the current [Flink 1 Common CI job](https://github.com/apache/paimon/actions/runs/36658431279/job/109707705933) was cancelled during “Build dependencies and run tests” after about 100 minutes, so `CI result` is still failing. The other selected engine and E2E jobs passed. Please obtain a completed green Flink 1 Common check before merging; the local focused tests do not establish that the cancelled full suite completed.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org