[GitHub] [druid] gianm commented on pull request #10267: DruidInputSource: Fix issues in column projection, timestamp handling.

[GitBox <gi...@apache.org>]

gianm commented on pull request #10267:
URL: https://github.com/apache/druid/pull/10267#issuecomment-672522776


   > (1) and (3) are breaking changes.
   
   Fwiw, on this one, I think the likelihood of (1) causing problems is low. It's a breaking change because if you were previously specifying a column as an input to one of your dimensionsSpec or aggregators, but then explicitly _not_ including in the input source's "dimensions" or "metrics" list, it'll now actually get read. Previously it'd be treated as null. The new behavior is better & less brittle but is different.
   
   (3) is still fairly low, but somewhat more likely. It's possible that someone had their timestampSpec set to something like `{"column": "__time", "format": "iso"}`, which is "wrong" but would have worked previously. Now it won't work: the timestamp will fail to parse (as it should, because it's not really in iso format).
   
   This patch as written just lets these things break, but we could cushion the fall, potentially:
   
   - For (1) we could throw an error in situations where people explicitly specify `dimensions` and `metrics` that don't match the computed inputs, informing them that these parameters no longer have an effect and asking them to remove them.
   - For (3) we could introduce a flag that causes an error to be thrown when you set the timestamp spec to anything other than `{"column": "__time", "format": "auto"}` or `{"column": "__time", "format": "millis"}`. Druid would invite you to set the flag, which would tell it "yes I really want the new, more correct behavior". That feature flag could be on or off by default.
   
   What do people think?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org