[GitHub] [incubator-pinot] jenniferdai opened a new pull request #4590: Disallowing multiple inputs in preprocess

[GitBox <gi...@apache.org>]

jenniferdai opened a new pull request #4590: Disallowing multiple inputs in preprocess
URL: https://github.com/apache/incubator-pinot/pull/4590
 
 
   * We cannot support this in preprocess because mapreduce takes complete control of the output path(s). In order to support this, we would need control to pipe the exact folders we receive as input to multiple outputs. While we can programmatically determine record by record what goes into each output path, this does not support our use case. Each folder is a separate "day," and frequently, our customers will have two dates in one file, due to timezone of data, so we are not able to distinguish what is "today's" vs. "tomorrow's" data by solely looking at the record.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org