[GitHub] [spark] d80tb7 commented on pull request #31399: [SPARK-34259][SQL] Don't attempt to parse file-based partitions as special timestamps

[GitBox <gi...@apache.org>]

d80tb7 commented on pull request #31399:
URL: https://github.com/apache/spark/pull/31399#issuecomment-770271817


   > I would create a dedicated date/timestamp formatters that don't recognize special values instead of checking of every input string whether it is a special value or not.
   > 
   Hi- I considered this approach but I was a bit put off as the code actually delegates creating the date/timestamp formatter to a factory which  branches off into multiple different classes depending on whether we're in legacy mode etc. We could update the factory such that it knows how to create formatters that don't recognize special values but that would be a more invasive change.  Let me know if that's the way you'd like to go and I'll raise a suitable PR.
   
   Also I know it's a technicality, but the change here  doesn't actually check every input string for special timestamps, just those that  successfully parse into a timestamp.
   
   > Could you elaborate more in PR description why special date/timestamp values in partition cause problems.
   
   Sure
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org