Re: [PR] [SPARK-33863][PYTHON] Respect session timezone in udf workers [spark]

["gaogaotiantian (via GitHub)" <gi...@apache.org>]

gaogaotiantian commented on PR #53161:
URL: https://github.com/apache/spark/pull/53161#issuecomment-3622866287

   I thought about it and I have an alternative proposal. We can add a conf to enable "strict mode" for timestamps. Where we always pair the aware timestamp with TimestampType and naive timestamp with TimestampNTZType. This is off by default but strict and correct when enabled.
   
   We have to hook this logic into the type conversion and I think the least intrusive way is to set it as a class variable of `TimestampType` - hook the change on conf change or something.
   
   In this way, when the user asks about the weird timestamp behavior, we can at least say - well the default config will never work properly, but you can try the strict mode. We also have a chance to gradually switch to strict mode in the future.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org