You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "vinishjail97 (via GitHub)" <gi...@apache.org> on 2024/03/14 18:42:12 UTC

[PR] Compute offsetRanges based on eventsPerPartition allocated in each range [hudi]

vinishjail97 opened a new pull request, #10869:
URL: https://github.com/apache/hudi/pull/10869

   ### Change Logs
   
   The current logic for computing offset ranges is leading to skews and negative offsets because of the way they are calculated. 
   https://github.com/apache/hudi/blob/master/hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KafkaOffsetGen.java#L144 
   
   Problems faced. 
   1. We are calculating eventsPerPartition based on available partitions that are not exhausted this can lead to skews where one partition handles only 1-10 messages and the remaining one handles 100K messages, the idea for minPartitions is to increase the parallelism and ensure that each spark task is reading approximately the same number of events. 
   2. remainingPartitions can become negative when finalRanges exceeds the size of minPartitions. 
   3. Complicated fork in code when minPartitions > toOffsetsMap, this is not required IMO and the default minPartitions can always fall back toOffsetsMap.size(), this takes care of situations when the partitions increase in kafka as well. 
   
   ```
             long remainingEvents = actualNumEvents - allocatedEvents;
             long remainingPartitions = toOffsetMap.size() - allocatedPartitionsThisLoop.size();
             // if need tp split into minPartitions, recalculate the remainingPartitions
             if (needSplitToMinPartitions) {
               remainingPartitions = minPartitions - finalRanges.size();
             }
             long eventsPerPartition = (long) Math.ceil((1.0 * remainingEvents) / remainingPartitions);
   ```
   
   New Approach
   1. Find eventsPerPartition which would be Math.max(1L, actualNumEvents / minPartitions);
   2. Keep computing offsetRanges unless allocatedEvents < actualNumEvents, compute them in a round-robin manner and keep the upper limit of eventsPerPartition messages for each range.
   3.  Return all the offsetRanges in the end after sorting them by partition 
   
   ### Impact
   
   No change in public API, the skew in spark task  partitions and bug related to negative remainingPartitions when consuming from Kafka is being fixed.
   
   ### Risk level (write none, low medium or high below)
   
   Medium
   
   ### Documentation Update
   
   None, the existing config KafkaSourceConfig.KAFKA_SOURCE_MIN_PARTITIONS is now being used the way it's supposed to be used. 
   
   ### Contributor's checklist
   
   - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [x] Change Logs and Impact were stated clearly
   - [x] Adequate tests were added if applicable
   - [ ] CI passed
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-7506] Compute offsetRanges based on eventsPerPartition allocated in each range [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10869:
URL: https://github.com/apache/hudi/pull/10869#issuecomment-1998450659

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "3369d9e8f41c77e3e5f0c371aea6884ca2d20555",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22915",
       "triggerID" : "3369d9e8f41c77e3e5f0c371aea6884ca2d20555",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 3369d9e8f41c77e3e5f0c371aea6884ca2d20555 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22915) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-7506] Compute offsetRanges based on eventsPerPartition allocated in each range [hudi]

Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan merged PR #10869:
URL: https://github.com/apache/hudi/pull/10869


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-7506] Compute offsetRanges based on eventsPerPartition allocated in each range [hudi]

Posted by "vinishjail97 (via GitHub)" <gi...@apache.org>.
vinishjail97 commented on code in PR #10869:
URL: https://github.com/apache/hudi/pull/10869#discussion_r1526598050


##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KafkaOffsetGen.java:
##########
@@ -129,63 +130,56 @@ public static OffsetRange[] computeOffsetRanges(Map<TopicPartition, Long> fromOf
           .toArray(new OffsetRange[toOffsetMap.size()]);
       LOG.debug("numEvents {}, minPartitions {}, ranges {}", numEvents, minPartitions, ranges);
 
-      boolean needSplitToMinPartitions = minPartitions > toOffsetMap.size();
-      long totalEvents = totalNewMessages(ranges);
-      long allocatedEvents = 0;
-      Set<Integer> exhaustedPartitions = new HashSet<>();
-      List<OffsetRange> finalRanges = new ArrayList<>();
       // choose the actualNumEvents with min(totalEvents, numEvents)
-      long actualNumEvents = Math.min(totalEvents, numEvents);
-
-      // keep going until we have events to allocate and partitions still not exhausted.
-      while (allocatedEvents < numEvents && exhaustedPartitions.size() < toOffsetMap.size()) {
-        // Allocate the remaining events to non-exhausted partitions, in round robin fashion
-        Set<Integer> allocatedPartitionsThisLoop = new HashSet<>(exhaustedPartitions);
-        for (int i = 0; i < ranges.length; i++) {
-          long remainingEvents = actualNumEvents - allocatedEvents;
-          long remainingPartitions = toOffsetMap.size() - allocatedPartitionsThisLoop.size();
-          // if need tp split into minPartitions, recalculate the remainingPartitions
-          if (needSplitToMinPartitions) {
-            remainingPartitions = minPartitions - finalRanges.size();
-          }
-          long eventsPerPartition = (long) Math.ceil((1.0 * remainingEvents) / remainingPartitions);
-
-          OffsetRange range = ranges[i];
-          if (exhaustedPartitions.contains(range.partition())) {
-            continue;
+      long actualNumEvents = Math.min(totalNewMessages(ranges), numEvents);
+      minPartitions = Math.max(minPartitions, toOffsetMap.size());
+      // Each OffsetRange computed will have maximum of eventsPerPartition,
+      // this ensures all ranges are evenly distributed and there's no skew in one particular range.
+      long eventsPerPartition = Math.max(1L, actualNumEvents / minPartitions);
+      long allocatedEvents = 0;
+      Map<TopicPartition, List<OffsetRange>> finalRanges = new HashMap<>();
+      // keep going until we have events to allocate.
+      while (allocatedEvents < actualNumEvents) {
+        // Allocate the remaining events in round-robin fashion.
+        for (OffsetRange range : ranges) {
+          // Compute startOffset.
+          long startOffset = range.fromOffset();
+          if (finalRanges.containsKey(range.topicPartition()) && !finalRanges.get(range.topicPartition()).isEmpty()) {
+            List<OffsetRange> offsetRangesForPartition = finalRanges.get(range.topicPartition());
+            startOffset = offsetRangesForPartition.get(offsetRangesForPartition.size() - 1).untilOffset();
           }
-
+          // Compute toOffset.
           long toOffset = -1L;
-          if (range.fromOffset() + eventsPerPartition > range.fromOffset()) {
-            toOffset = Math.min(range.untilOffset(), range.fromOffset() + eventsPerPartition);
+          if (startOffset + eventsPerPartition > startOffset) {
+            toOffset = Math.min(range.untilOffset(), startOffset + eventsPerPartition);
           } else {
             // handling Long overflow
             toOffset = range.untilOffset();
           }
-          if (toOffset == range.untilOffset()) {
-            exhaustedPartitions.add(range.partition());
-          }
-          // We need recompute toOffset if we have allocatedEvents are more than actualNumEvents.
-          long totalAllocatedEvents = allocatedEvents + (toOffset - range.fromOffset());
+          // We need recompute toOffset if we allocatedEvents are more than actualNumEvents.
+          long totalAllocatedEvents = allocatedEvents + (toOffset - startOffset);
           if (totalAllocatedEvents > actualNumEvents) {
             long offsetsToAdd = Math.min(eventsPerPartition, (actualNumEvents - allocatedEvents));
-            toOffset = Math.min(range.untilOffset(), range.fromOffset() + offsetsToAdd);
+            toOffset = Math.min(range.untilOffset(), startOffset + offsetsToAdd);

Review Comment:
   LGTM, the break statement simplifies things.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-7506] Compute offsetRanges based on eventsPerPartition allocated in each range [hudi]

Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #10869:
URL: https://github.com/apache/hudi/pull/10869#discussion_r1525659089


##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KafkaOffsetGen.java:
##########
@@ -129,63 +130,56 @@ public static OffsetRange[] computeOffsetRanges(Map<TopicPartition, Long> fromOf
           .toArray(new OffsetRange[toOffsetMap.size()]);
       LOG.debug("numEvents {}, minPartitions {}, ranges {}", numEvents, minPartitions, ranges);
 
-      boolean needSplitToMinPartitions = minPartitions > toOffsetMap.size();
-      long totalEvents = totalNewMessages(ranges);
-      long allocatedEvents = 0;
-      Set<Integer> exhaustedPartitions = new HashSet<>();
-      List<OffsetRange> finalRanges = new ArrayList<>();
       // choose the actualNumEvents with min(totalEvents, numEvents)
-      long actualNumEvents = Math.min(totalEvents, numEvents);
-
-      // keep going until we have events to allocate and partitions still not exhausted.
-      while (allocatedEvents < numEvents && exhaustedPartitions.size() < toOffsetMap.size()) {
-        // Allocate the remaining events to non-exhausted partitions, in round robin fashion
-        Set<Integer> allocatedPartitionsThisLoop = new HashSet<>(exhaustedPartitions);
-        for (int i = 0; i < ranges.length; i++) {
-          long remainingEvents = actualNumEvents - allocatedEvents;
-          long remainingPartitions = toOffsetMap.size() - allocatedPartitionsThisLoop.size();
-          // if need tp split into minPartitions, recalculate the remainingPartitions
-          if (needSplitToMinPartitions) {
-            remainingPartitions = minPartitions - finalRanges.size();
-          }
-          long eventsPerPartition = (long) Math.ceil((1.0 * remainingEvents) / remainingPartitions);
-
-          OffsetRange range = ranges[i];
-          if (exhaustedPartitions.contains(range.partition())) {
-            continue;
+      long actualNumEvents = Math.min(totalNewMessages(ranges), numEvents);
+      minPartitions = Math.max(minPartitions, toOffsetMap.size());
+      // Each OffsetRange computed will have maximum of eventsPerPartition,
+      // this ensures all ranges are evenly distributed and there's no skew in one particular range.
+      long eventsPerPartition = Math.max(1L, actualNumEvents / minPartitions);
+      long allocatedEvents = 0;
+      Map<TopicPartition, List<OffsetRange>> finalRanges = new HashMap<>();
+      // keep going until we have events to allocate.
+      while (allocatedEvents < actualNumEvents) {
+        // Allocate the remaining events in round-robin fashion.
+        for (OffsetRange range : ranges) {
+          // Compute startOffset.
+          long startOffset = range.fromOffset();
+          if (finalRanges.containsKey(range.topicPartition()) && !finalRanges.get(range.topicPartition()).isEmpty()) {
+            List<OffsetRange> offsetRangesForPartition = finalRanges.get(range.topicPartition());
+            startOffset = offsetRangesForPartition.get(offsetRangesForPartition.size() - 1).untilOffset();

Review Comment:
   should we maintain a hashmap of partition -> latestEndOffset 
   rather than going over finalRanges and computing each time.
   reads simpler. 



##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KafkaOffsetGen.java:
##########
@@ -129,63 +130,56 @@ public static OffsetRange[] computeOffsetRanges(Map<TopicPartition, Long> fromOf
           .toArray(new OffsetRange[toOffsetMap.size()]);
       LOG.debug("numEvents {}, minPartitions {}, ranges {}", numEvents, minPartitions, ranges);
 
-      boolean needSplitToMinPartitions = minPartitions > toOffsetMap.size();
-      long totalEvents = totalNewMessages(ranges);
-      long allocatedEvents = 0;
-      Set<Integer> exhaustedPartitions = new HashSet<>();
-      List<OffsetRange> finalRanges = new ArrayList<>();
       // choose the actualNumEvents with min(totalEvents, numEvents)
-      long actualNumEvents = Math.min(totalEvents, numEvents);
-
-      // keep going until we have events to allocate and partitions still not exhausted.
-      while (allocatedEvents < numEvents && exhaustedPartitions.size() < toOffsetMap.size()) {
-        // Allocate the remaining events to non-exhausted partitions, in round robin fashion
-        Set<Integer> allocatedPartitionsThisLoop = new HashSet<>(exhaustedPartitions);
-        for (int i = 0; i < ranges.length; i++) {
-          long remainingEvents = actualNumEvents - allocatedEvents;
-          long remainingPartitions = toOffsetMap.size() - allocatedPartitionsThisLoop.size();
-          // if need tp split into minPartitions, recalculate the remainingPartitions
-          if (needSplitToMinPartitions) {
-            remainingPartitions = minPartitions - finalRanges.size();
-          }
-          long eventsPerPartition = (long) Math.ceil((1.0 * remainingEvents) / remainingPartitions);
-
-          OffsetRange range = ranges[i];
-          if (exhaustedPartitions.contains(range.partition())) {
-            continue;
+      long actualNumEvents = Math.min(totalNewMessages(ranges), numEvents);
+      minPartitions = Math.max(minPartitions, toOffsetMap.size());
+      // Each OffsetRange computed will have maximum of eventsPerPartition,
+      // this ensures all ranges are evenly distributed and there's no skew in one particular range.
+      long eventsPerPartition = Math.max(1L, actualNumEvents / minPartitions);
+      long allocatedEvents = 0;
+      Map<TopicPartition, List<OffsetRange>> finalRanges = new HashMap<>();
+      // keep going until we have events to allocate.
+      while (allocatedEvents < actualNumEvents) {
+        // Allocate the remaining events in round-robin fashion.
+        for (OffsetRange range : ranges) {
+          // Compute startOffset.
+          long startOffset = range.fromOffset();
+          if (finalRanges.containsKey(range.topicPartition()) && !finalRanges.get(range.topicPartition()).isEmpty()) {
+            List<OffsetRange> offsetRangesForPartition = finalRanges.get(range.topicPartition());
+            startOffset = offsetRangesForPartition.get(offsetRangesForPartition.size() - 1).untilOffset();
           }
-
+          // Compute toOffset.
           long toOffset = -1L;
-          if (range.fromOffset() + eventsPerPartition > range.fromOffset()) {
-            toOffset = Math.min(range.untilOffset(), range.fromOffset() + eventsPerPartition);
+          if (startOffset + eventsPerPartition > startOffset) {
+            toOffset = Math.min(range.untilOffset(), startOffset + eventsPerPartition);
           } else {
             // handling Long overflow
             toOffset = range.untilOffset();
           }
-          if (toOffset == range.untilOffset()) {
-            exhaustedPartitions.add(range.partition());
-          }
-          // We need recompute toOffset if we have allocatedEvents are more than actualNumEvents.
-          long totalAllocatedEvents = allocatedEvents + (toOffset - range.fromOffset());
+          // We need recompute toOffset if we allocatedEvents are more than actualNumEvents.
+          long totalAllocatedEvents = allocatedEvents + (toOffset - startOffset);
           if (totalAllocatedEvents > actualNumEvents) {
             long offsetsToAdd = Math.min(eventsPerPartition, (actualNumEvents - allocatedEvents));
-            toOffset = Math.min(range.untilOffset(), range.fromOffset() + offsetsToAdd);
+            toOffset = Math.min(range.untilOffset(), startOffset + offsetsToAdd);

Review Comment:
   L160 to 164 seems complex. 
   I tried simplifying this entire code block. can you check it out. 
   ```
        // choose the actualNumEvents with min(totalEvents, numEvents)
         long actualNumEvents = Math.min(totalNewMessages(ranges), numEvents);
         minPartitions = Math.max(minPartitions, toOffsetMap.size());
         // Each OffsetRange computed will have maximum of eventsPerPartition,
         // this ensures all ranges are evenly distributed and there's no skew in one particular range.
         long eventsPerPartition = Math.max(1L, actualNumEvents / minPartitions);
         long allocatedEvents = 0;
         Map<TopicPartition, List<OffsetRange>> finalRanges = new HashMap<>();
         Map<TopicPartition, Long> partitionToAllocatedOffset = new HashMap<>();
         // keep going until we have events to allocate.
         while (allocatedEvents < actualNumEvents) {
           // Allocate the remaining events in round-robin fashion.
           for (OffsetRange range : ranges) {
             // if we have already allocated required no of events, exit
             if (allocatedEvents == actualNumEvents) {
               break;
             }
             // Compute startOffset.
             long startOffset = range.fromOffset();
             if (partitionToAllocatedOffset.containsKey(range.topicPartition())) {
               startOffset = partitionToAllocatedOffset.get(range.topicPartition());
             }
   
             // for last bucket, we may not have full eventsPerPartition msgs.
             long eventsForThisPartition = Math.min(eventsPerPartition, (actualNumEvents - allocatedEvents));
             // Compute toOffset.
             long toOffset = -1L;
             if (startOffset + eventsForThisPartition > startOffset) {
               toOffset = Math.min(range.untilOffset(), startOffset + eventsForThisPartition);
             } else {
               // handling Long overflow
               toOffset = range.untilOffset();
             }
             allocatedEvents += toOffset - startOffset;
             OffsetRange thisRange = OffsetRange.create(range.topicPartition(), startOffset, toOffset);
             // Add the offsetRange(startOffset,toOffset) to finalRanges.
             if (!finalRanges.containsKey(range.topicPartition())) {
               finalRanges.put(range.topicPartition(), new ArrayList<>(Collections.singleton(thisRange)));
               partitionToAllocatedOffset.put(range.topicPartition(), thisRange.untilOffset());
             } else if (toOffset > startOffset) {
               finalRanges.get(range.topicPartition()).add(thisRange);
               partitionToAllocatedOffset.put(range.topicPartition(), thisRange.untilOffset());
             }
            }
         }
   ```
   
   All the existing tests succeed. and is simpler to read and maintain.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-7506] Compute offsetRanges based on eventsPerPartition allocated in each range [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10869:
URL: https://github.com/apache/hudi/pull/10869#issuecomment-1998111406

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "3369d9e8f41c77e3e5f0c371aea6884ca2d20555",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "3369d9e8f41c77e3e5f0c371aea6884ca2d20555",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 3369d9e8f41c77e3e5f0c371aea6884ca2d20555 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-7506] Compute offsetRanges based on eventsPerPartition allocated in each range [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10869:
URL: https://github.com/apache/hudi/pull/10869#issuecomment-2000178004

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "3369d9e8f41c77e3e5f0c371aea6884ca2d20555",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22915",
       "triggerID" : "3369d9e8f41c77e3e5f0c371aea6884ca2d20555",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e5463d465400005260cc4f388ee1ad2d419384f9",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22926",
       "triggerID" : "e5463d465400005260cc4f388ee1ad2d419384f9",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 3369d9e8f41c77e3e5f0c371aea6884ca2d20555 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22915) 
   * e5463d465400005260cc4f388ee1ad2d419384f9 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22926) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-7506] Compute offsetRanges based on eventsPerPartition allocated in each range [hudi]

Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on PR #10869:
URL: https://github.com/apache/hudi/pull/10869#issuecomment-2000310516

   <img width="1210" alt="image" src="https://github.com/apache/hudi/assets/513218/d137b7db-b4b0-4d28-bedc-19101e63cbbc">
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-7506] Compute offsetRanges based on eventsPerPartition allocated in each range [hudi]

Posted by "vinishjail97 (via GitHub)" <gi...@apache.org>.
vinishjail97 commented on code in PR #10869:
URL: https://github.com/apache/hudi/pull/10869#discussion_r1526605929


##########
hudi-utilities/src/test/java/org/apache/hudi/utilities/sources/helpers/TestKafkaOffsetGen.java:
##########
@@ -140,11 +140,13 @@ public void testGetNextOffsetRangesFromMultiplePartitions() {
     testUtils.sendMessages(testTopicName, Helpers.jsonifyRecords(dataGenerator.generateInserts("000", 1000)));
     KafkaOffsetGen kafkaOffsetGen = new KafkaOffsetGen(getConsumerConfigs("earliest", "string"));
     OffsetRange[] nextOffsetRanges = kafkaOffsetGen.getNextOffsetRanges(Option.empty(), 499, metrics);
-    assertEquals(2, nextOffsetRanges.length);
+    assertEquals(3, nextOffsetRanges.length);

Review Comment:
   The eventsPerPartition for 499 messages and 2 partitions is 249 messages per offsetRange, To fill up 499 messages 3(249 + 249 + 1) offset ranges. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-7506] Compute offsetRanges based on eventsPerPartition allocated in each range [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10869:
URL: https://github.com/apache/hudi/pull/10869#issuecomment-2000167185

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "3369d9e8f41c77e3e5f0c371aea6884ca2d20555",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22915",
       "triggerID" : "3369d9e8f41c77e3e5f0c371aea6884ca2d20555",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e5463d465400005260cc4f388ee1ad2d419384f9",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "e5463d465400005260cc4f388ee1ad2d419384f9",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 3369d9e8f41c77e3e5f0c371aea6884ca2d20555 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22915) 
   * e5463d465400005260cc4f388ee1ad2d419384f9 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-7506] Compute offsetRanges based on eventsPerPartition allocated in each range [hudi]

Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #10869:
URL: https://github.com/apache/hudi/pull/10869#discussion_r1526290033


##########
hudi-utilities/src/test/java/org/apache/hudi/utilities/sources/helpers/TestKafkaOffsetGen.java:
##########
@@ -140,11 +140,13 @@ public void testGetNextOffsetRangesFromMultiplePartitions() {
     testUtils.sendMessages(testTopicName, Helpers.jsonifyRecords(dataGenerator.generateInserts("000", 1000)));
     KafkaOffsetGen kafkaOffsetGen = new KafkaOffsetGen(getConsumerConfigs("earliest", "string"));
     OffsetRange[] nextOffsetRanges = kafkaOffsetGen.getNextOffsetRanges(Option.empty(), 499, metrics);
-    assertEquals(2, nextOffsetRanges.length);
+    assertEquals(3, nextOffsetRanges.length);

Review Comment:
   why should the expected value change?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-7506] Compute offsetRanges based on eventsPerPartition allocated in each range [hudi]

Posted by "vinishjail97 (via GitHub)" <gi...@apache.org>.
vinishjail97 commented on code in PR #10869:
URL: https://github.com/apache/hudi/pull/10869#discussion_r1525369671


##########
hudi-utilities/src/test/java/org/apache/hudi/utilities/sources/helpers/TestCheckpointUtils.java:
##########
@@ -104,26 +105,35 @@ public void testComputeOffsetRangesWithoutMinPartitions() {
     ranges = CheckpointUtils.computeOffsetRanges(makeOffsetMap(new int[] {0, 1}, new long[] {200000, 250000}),
         makeOffsetMap(new int[] {0, 1, 2}, new long[] {200010, 350000, 10000}), 100000, 0);
     assertEquals(100000, CheckpointUtils.totalNewMessages(ranges));
+    assertEquals(5, ranges.length);
+    assertEquals(0, ranges[0].partition());
     assertEquals(10, ranges[0].count());
-    assertEquals(89990, ranges[1].count());
-    assertEquals(10000, ranges[2].count());

Review Comment:
   The skew caused by the current approach the different can be more if the last partition to be exhausted has the highest number of backlog messages to be consumed. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-7506] Compute offsetRanges based on eventsPerPartition allocated in each range [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10869:
URL: https://github.com/apache/hudi/pull/10869#issuecomment-1998139848

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "3369d9e8f41c77e3e5f0c371aea6884ca2d20555",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22915",
       "triggerID" : "3369d9e8f41c77e3e5f0c371aea6884ca2d20555",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 3369d9e8f41c77e3e5f0c371aea6884ca2d20555 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22915) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-7506] Compute offsetRanges based on eventsPerPartition allocated in each range [hudi]

Posted by "vinishjail97 (via GitHub)" <gi...@apache.org>.
vinishjail97 commented on code in PR #10869:
URL: https://github.com/apache/hudi/pull/10869#discussion_r1526598050


##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KafkaOffsetGen.java:
##########
@@ -129,63 +130,56 @@ public static OffsetRange[] computeOffsetRanges(Map<TopicPartition, Long> fromOf
           .toArray(new OffsetRange[toOffsetMap.size()]);
       LOG.debug("numEvents {}, minPartitions {}, ranges {}", numEvents, minPartitions, ranges);
 
-      boolean needSplitToMinPartitions = minPartitions > toOffsetMap.size();
-      long totalEvents = totalNewMessages(ranges);
-      long allocatedEvents = 0;
-      Set<Integer> exhaustedPartitions = new HashSet<>();
-      List<OffsetRange> finalRanges = new ArrayList<>();
       // choose the actualNumEvents with min(totalEvents, numEvents)
-      long actualNumEvents = Math.min(totalEvents, numEvents);
-
-      // keep going until we have events to allocate and partitions still not exhausted.
-      while (allocatedEvents < numEvents && exhaustedPartitions.size() < toOffsetMap.size()) {
-        // Allocate the remaining events to non-exhausted partitions, in round robin fashion
-        Set<Integer> allocatedPartitionsThisLoop = new HashSet<>(exhaustedPartitions);
-        for (int i = 0; i < ranges.length; i++) {
-          long remainingEvents = actualNumEvents - allocatedEvents;
-          long remainingPartitions = toOffsetMap.size() - allocatedPartitionsThisLoop.size();
-          // if need tp split into minPartitions, recalculate the remainingPartitions
-          if (needSplitToMinPartitions) {
-            remainingPartitions = minPartitions - finalRanges.size();
-          }
-          long eventsPerPartition = (long) Math.ceil((1.0 * remainingEvents) / remainingPartitions);
-
-          OffsetRange range = ranges[i];
-          if (exhaustedPartitions.contains(range.partition())) {
-            continue;
+      long actualNumEvents = Math.min(totalNewMessages(ranges), numEvents);
+      minPartitions = Math.max(minPartitions, toOffsetMap.size());
+      // Each OffsetRange computed will have maximum of eventsPerPartition,
+      // this ensures all ranges are evenly distributed and there's no skew in one particular range.
+      long eventsPerPartition = Math.max(1L, actualNumEvents / minPartitions);
+      long allocatedEvents = 0;
+      Map<TopicPartition, List<OffsetRange>> finalRanges = new HashMap<>();
+      // keep going until we have events to allocate.
+      while (allocatedEvents < actualNumEvents) {
+        // Allocate the remaining events in round-robin fashion.
+        for (OffsetRange range : ranges) {
+          // Compute startOffset.
+          long startOffset = range.fromOffset();
+          if (finalRanges.containsKey(range.topicPartition()) && !finalRanges.get(range.topicPartition()).isEmpty()) {
+            List<OffsetRange> offsetRangesForPartition = finalRanges.get(range.topicPartition());
+            startOffset = offsetRangesForPartition.get(offsetRangesForPartition.size() - 1).untilOffset();
           }
-
+          // Compute toOffset.
           long toOffset = -1L;
-          if (range.fromOffset() + eventsPerPartition > range.fromOffset()) {
-            toOffset = Math.min(range.untilOffset(), range.fromOffset() + eventsPerPartition);
+          if (startOffset + eventsPerPartition > startOffset) {
+            toOffset = Math.min(range.untilOffset(), startOffset + eventsPerPartition);
           } else {
             // handling Long overflow
             toOffset = range.untilOffset();
           }
-          if (toOffset == range.untilOffset()) {
-            exhaustedPartitions.add(range.partition());
-          }
-          // We need recompute toOffset if we have allocatedEvents are more than actualNumEvents.
-          long totalAllocatedEvents = allocatedEvents + (toOffset - range.fromOffset());
+          // We need recompute toOffset if we allocatedEvents are more than actualNumEvents.
+          long totalAllocatedEvents = allocatedEvents + (toOffset - startOffset);
           if (totalAllocatedEvents > actualNumEvents) {
             long offsetsToAdd = Math.min(eventsPerPartition, (actualNumEvents - allocatedEvents));
-            toOffset = Math.min(range.untilOffset(), range.fromOffset() + offsetsToAdd);
+            toOffset = Math.min(range.untilOffset(), startOffset + offsetsToAdd);

Review Comment:
   LGTM, the break statement simplifies things made the change. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-7506] Compute offsetRanges based on eventsPerPartition allocated in each range [hudi]

Posted by "vinishjail97 (via GitHub)" <gi...@apache.org>.
vinishjail97 commented on code in PR #10869:
URL: https://github.com/apache/hudi/pull/10869#discussion_r1526598398


##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KafkaOffsetGen.java:
##########
@@ -129,63 +130,56 @@ public static OffsetRange[] computeOffsetRanges(Map<TopicPartition, Long> fromOf
           .toArray(new OffsetRange[toOffsetMap.size()]);
       LOG.debug("numEvents {}, minPartitions {}, ranges {}", numEvents, minPartitions, ranges);
 
-      boolean needSplitToMinPartitions = minPartitions > toOffsetMap.size();
-      long totalEvents = totalNewMessages(ranges);
-      long allocatedEvents = 0;
-      Set<Integer> exhaustedPartitions = new HashSet<>();
-      List<OffsetRange> finalRanges = new ArrayList<>();
       // choose the actualNumEvents with min(totalEvents, numEvents)
-      long actualNumEvents = Math.min(totalEvents, numEvents);
-
-      // keep going until we have events to allocate and partitions still not exhausted.
-      while (allocatedEvents < numEvents && exhaustedPartitions.size() < toOffsetMap.size()) {
-        // Allocate the remaining events to non-exhausted partitions, in round robin fashion
-        Set<Integer> allocatedPartitionsThisLoop = new HashSet<>(exhaustedPartitions);
-        for (int i = 0; i < ranges.length; i++) {
-          long remainingEvents = actualNumEvents - allocatedEvents;
-          long remainingPartitions = toOffsetMap.size() - allocatedPartitionsThisLoop.size();
-          // if need tp split into minPartitions, recalculate the remainingPartitions
-          if (needSplitToMinPartitions) {
-            remainingPartitions = minPartitions - finalRanges.size();
-          }
-          long eventsPerPartition = (long) Math.ceil((1.0 * remainingEvents) / remainingPartitions);
-
-          OffsetRange range = ranges[i];
-          if (exhaustedPartitions.contains(range.partition())) {
-            continue;
+      long actualNumEvents = Math.min(totalNewMessages(ranges), numEvents);
+      minPartitions = Math.max(minPartitions, toOffsetMap.size());
+      // Each OffsetRange computed will have maximum of eventsPerPartition,
+      // this ensures all ranges are evenly distributed and there's no skew in one particular range.
+      long eventsPerPartition = Math.max(1L, actualNumEvents / minPartitions);
+      long allocatedEvents = 0;
+      Map<TopicPartition, List<OffsetRange>> finalRanges = new HashMap<>();
+      // keep going until we have events to allocate.
+      while (allocatedEvents < actualNumEvents) {
+        // Allocate the remaining events in round-robin fashion.
+        for (OffsetRange range : ranges) {
+          // Compute startOffset.
+          long startOffset = range.fromOffset();
+          if (finalRanges.containsKey(range.topicPartition()) && !finalRanges.get(range.topicPartition()).isEmpty()) {
+            List<OffsetRange> offsetRangesForPartition = finalRanges.get(range.topicPartition());
+            startOffset = offsetRangesForPartition.get(offsetRangesForPartition.size() - 1).untilOffset();

Review Comment:
   Addressed as mentioned in the second comment.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org