[GitHub] [pulsar] merlimat commented on issue #4320: Change default backlog quota retention policy to consumer_backlog_eviction

[GitBox <gi...@apache.org>]

merlimat commented on issue #4320: Change default backlog quota retention policy to consumer_backlog_eviction
URL: https://github.com/apache/pulsar/pull/4320#issuecomment-495320496
 
 
   @codelipenghui 
   
   > If I didn’t understand it wrong, this is just to guarantee that the data that has been written is not be lost. If we look at the user's entire system, pulsar is the important part, serving real-time streaming data. If it becomes unwritable, this is also data loss for the user.
   
   Of course if the system is not-writable, the application will be impacted, though the main difference is that what was "acked" before is not lost.
   
   Applications can apply fallback strategies when a downstream system is unavailable, from the simplest (fail user request) to more complex (degraded functioning mode), but they won't be able to know if "acked" data was dropped by the system. There would be no immediate way to communicate that to users, or to apply other strategies.
   
   > The core of the problem is how the user should handle when the backlog has exceeded the threshold, delete old data or reject new data. For streaming data platforms, i think rejecting new data is basically unacceptable. Users can use the retention mechanism to confirm how long they retain data.
   
   The default option can be set in broker.conf and overridden per namespace, to adjust to needs.
   
   > For the backlog, this is for a subscriber. There will be one subscriber's backlog cause all new data to be rejected.
   
   Without information around what's the importance of the subscriptions, it's difficult to generalize the judgement of which subscription should retain data.
   
   Subscriptions are a way to explicitly retain data in the system.
   
   > If the default configuration is inclined to not lost data. reject new data should also be considered.
   
   That's what the default setting is.
   
   
   @srkukarni 
   
   > I believe that streaming systems usually prefer writable system. However at the same time I believe queuing systems prefer non-loss of any transactions. Since Pulsar supports both, this is a tricky issue.
   
   That also comes with the dichotomy between Consumer vs Reader. 
   In streaming mode, to me, a Reader makes more sense in that it does imply data retention, therefore it would never incur in a backlog quota violation.
   
   @codelipenghui 
   
   > If do not consider resource limitations, not limit the backlog by default can match both two . My suggestion is to prefer the configuration with the least loss to the user as the default configuration.
   
   Yes, the default quota setting is very arbitrary (and very low). I'd be fine to disable backlog quota check by default. It will simplify many "gotchas", though that won't remove the reality that disk space is finite (even if most likely > 10GB),  and applications need to reason about what to do when the limit is reached.
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services