[GitHub] MaxGekk commented on a change in pull request #23662: [SPARK-26740][SQL] Make statistics of timestamp/date columns independent from system time zones

[GitBox <gi...@apache.org>]

MaxGekk commented on a change in pull request #23662: [SPARK-26740][SQL] Make statistics of timestamp/date columns independent from system time zones
URL: https://github.com/apache/spark/pull/23662#discussion_r251357165
 
 

 ##########
 File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/interface.scala
 ##########
 @@ -501,9 +501,8 @@ object CatalogColumnStat extends Logging {
    */
   def toExternalString(v: Any, colName: String, dataType: DataType): String = {
     val externalValue = dataType match {
-      case DateType => DateTimeUtils.toJavaDate(v.asInstanceOf[Int])
-      case TimestampType => DateTimeUtils.toJavaTimestamp(v.asInstanceOf[Long])
-      case BooleanType | _: IntegralType | FloatType | DoubleType => v
+      case BooleanType | _: IntegralType | FloatType | DoubleType |
+           TimestampType | DateType => v
 
 Review comment:
   > If we do this, then the stats string will not be human-readable.
   
   Is it necessary to be like human-readable? 
   
   > Shall we still use a timestamp/date format, but put the timezone at the end?
   
   If I add timezone at the end, I am not sure the stored text can be more readable from meaning perspective. For example, you have `2019-01-28 01:00:00` in your time zone `UTC+8` but when I see that I have to subtract 8 hours to get real timestamp. Taking into account that it goes over the end of the day. 
   
   Unix timestamp can be easily converted to text, just google `unix time` - there are a lot of websites. 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org