[GitHub] [gobblin] sv2000 commented on a diff in pull request #3515: [GOBBLIN-1654] Add capacity floor to avoid aggressively requesting resource and small files.

[GitBox <gi...@apache.org>]

sv2000 commented on code in PR #3515:
URL: https://github.com/apache/gobblin/pull/3515#discussion_r884023049


##########
gobblin-modules/gobblin-kafka-common/src/main/java/org/apache/gobblin/source/extractor/extract/kafka/workunit/packer/KafkaTopicGroupingWorkUnitPacker.java:
##########
@@ -79,6 +80,13 @@ public class KafkaTopicGroupingWorkUnitPacker extends KafkaWorkUnitPacker {
   public static final String CONTAINER_CAPACITY_KEY = GOBBLIN_KAFKA_PREFIX + "streaming.containerCapacity";
   public static final double DEFAULT_CONTAINER_CAPACITY = 10;
 
+  // minimum container capacity to avoid bad topic schema causing us to request resources aggressively
+  public static final String MINIMUM_CONTAINER_CAPACITY = GOBBLIN_KAFKA_PREFIX + "streaming.minimum.containerCapacity";
+  public static final double DEFAULT_MINIMUM_CONTAINER_CAPACITY = 1;
+  public static final String BAD_TOPIC_PARTITION_WITH_LOW_CAPACITY_EVENT_NAME = "badTopicPartitionWithLowCapacity";

Review Comment:
   badTopicPartitionWithLowCapacity -> topicPartitionWithLowCapacity.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org