[GitHub] [spark] alf239 commented on pull request #31399: [SPARK-34259][SQL] Don't attempt to parse file-based partitions as special timestamps

[GitBox <gi...@apache.org>]

alf239 commented on pull request #31399:
URL: https://github.com/apache/spark/pull/31399#issuecomment-770285210


   The `TODAY` approach is imaginable indeed; I think the root problem is not the magic words as a timestamp, but rather the fact that a string gets reinterpreted as a timestamp, and even if we later consider that the type is a string after all, we just reinterpret the dates instead.
   
   So for example if you have (say) a task list, with the tasks split into `NOW`, `TOMORROW`, `LATER`, and `MAYBE`, the current implementation would convert `NOW` into a timestamp, and if there are multiple parquet files in the partition, that timestamp will be different for different files — that's where I don't quite believe in the `TODAY` approach: imagine a job that happens to run just on the midnight, reading a few parquet files as the clock strike 12: a single partition will be split at a random point.
   
   Moreover, let's say it treats `NOW` as 2021-01-30 21:41:33.945. It goes on to `LATER`, fails to parse it as a date, and decides that the column is string after all. But instead of falling back to `NOW` as one would expect for a string, it recasts the timestamp to string, and you end up with `2021-01-30 21:41:33.945`, something for `TOMORROW` (didn't try), `LATER`, and `MAYBE` which is clearly not an intended behaviour: those were never dates, and the values would be expected to stay stable. 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org