You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@gobblin.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2022/06/01 21:40:00 UTC

[jira] [Work logged] (GOBBLIN-1654) Add capacity floor to avoid aggressively requesting resource and small files.

     [ https://issues.apache.org/jira/browse/GOBBLIN-1654?focusedWorklogId=777182&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-777182 ]

ASF GitHub Bot logged work on GOBBLIN-1654:
-------------------------------------------

                Author: ASF GitHub Bot
            Created on: 01/Jun/22 21:39
            Start Date: 01/Jun/22 21:39
    Worklog Time Spent: 10m 
      Work Description: sv2000 merged PR #3515:
URL: https://github.com/apache/gobblin/pull/3515




Issue Time Tracking
-------------------

    Worklog Id:     (was: 777182)
    Time Spent: 40m  (was: 0.5h)

> Add capacity floor to avoid aggressively requesting resource and small files.
> -----------------------------------------------------------------------------
>
>                 Key: GOBBLIN-1654
>                 URL: https://issues.apache.org/jira/browse/GOBBLIN-1654
>             Project: Apache Gobblin
>          Issue Type: Improvement
>            Reporter: Zihan Li
>            Priority: Major
>          Time Spent: 40m
>  Remaining Estimate: 0h
>
> Now we calculate the capacity based on the consumer rate during peak hour, but if there are records with super bad schema during that time, our consumer rate will be super low and even after we catch up, we will not be able to release the resources because of the low consumer rate. So want provide a config to set the minimum consumer rate and avoid abusing resources and small files



--
This message was sent by Atlassian Jira
(v8.20.7#820007)