[GitHub] [incubator-pinot] npawar commented on issue #4345: Config to make realtime non-winner servers download the segment instead of build

[GitBox <gi...@apache.org>]

npawar commented on issue #4345: Config to make realtime non-winner servers download the segment instead of build
URL: https://github.com/apache/incubator-pinot/issues/4345#issuecomment-504241835
 
 
   Some options we are considering:
   1. A table level config
   ```
   segmentsConfig: {
     downloadConfig: {
       mode: DEFAULT or ALWAYS_DOWNLOAD or ALWAYS_BUILD or AUTOTUNE
      // additional config, such as replicas percentage or thresholds, build times, segment size etc
     }
   }
   ```
   The controller will read this config, and accordingly decide the response to send to a replica. The response state will not change. We can add a `boolean` field in the SegmentCompletionProtocol.Response called `download`. We can set this in the Response from SegmentCompletionProtocol.SegmentCompletionFSM::segmentConsumed. This can be as simple as reading table config and setting the boolean to begin with. Later on, we can evolve this to reading more signals, and sending separate responses to each replica
   2. Controller level config, with list of tables to enable
   The implementation details will be similar as option 1. Only difference will be how the config is fetched. We will not be able to add table specific fine tuning if we need to
   3. Server level config
   Boolean to enable "always download" on the server. This can be read directly in the PartitionConsumer::run method, in the KEEP case, and we will accordingly avoid building segment.
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org