[GitHub] [incubator-pinot] jackjlli commented on pull request #6107: Making pushType non-mandatory

[GitBox <gi...@apache.org>]

jackjlli commented on pull request #6107:
URL: https://github.com/apache/incubator-pinot/pull/6107#issuecomment-703921970


   The config `segmentPushType` is needed at least in the following scenarios:
   1. Retention manager needs it to purge segments: https://github.com/apache/incubator-pinot/blob/9929dad2e804f525b8ace925e859e07c6fdee9da/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/retention/RetentionManager.java#L104
   2. `TableRetentionValidator` gets called https://github.com/apache/incubator-pinot/blob/fb043dab7d5fef3ef726896d2f53e45a2fd69f22/pinot-controller/src/main/java/org/apache/pinot/controller/util/TableRetentionValidator.java#L112
   3. Segment creation Hadoop/Spark job requires it to validate the time column: https://github.com/apache/incubator-pinot/blob/fb043dab7d5fef3ef726896d2f53e45a2fd69f22/pinot-plugins/pinot-batch-ingestion/v0_deprecated/pinot-hadoop/src/main/java/org/apache/pinot/hadoop/job/HadoopSegmentCreationJob.java#L145
   4. Segment creation job requires it to generate the segment name: https://github.com/apache/incubator-pinot/blob/540853d786c9e4b4bb27bf4bbc6fa73837ff1bc3/pinot-core/src/main/java/org/apache/pinot/core/segment/name/NormalizedDateSegmentNameGenerator.java#L48
   
   So if you remove the check, you'd also need to adjust those logic to avoid breaking the current behaviors.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org