[GitHub] [incubator-iceberg] wangmiao1981 commented on issue #475: TimestampWriter isn't being used

[GitBox <gi...@apache.org>]

wangmiao1981 commented on issue #475: TimestampWriter isn't being used
URL: https://github.com/apache/incubator-iceberg/issues/475#issuecomment-607658305
 
 
   @rdblue Just hit the same issue. In TypeToMessageType.java, 
   https://github.com/apache/incubator-iceberg/blob/master/parquet/src/main/java/org/apache/iceberg/parquet/TypeToMessageType.java#L82, calls https://github.com/apache/incubator-iceberg/blob/master/parquet/src/main/java/org/apache/iceberg/parquet/TypeToMessageType.java#L133
   It can't store the `shouldAdjustToUTC ` field.  This is the reason it can't call the correct writer. So it fails in the `if` branch, https://github.com/apache/incubator-iceberg/blob/master/parquet/src/main/java/org/apache/iceberg/parquet/Parquet.java#L212
   
   There are few other existing tests pass, which go to https://github.com/apache/incubator-iceberg/blob/master/parquet/src/main/java/org/apache/iceberg/parquet/Parquet.java#L235, which ignores the boolean flag and write it as Timestamp.withZone(). So it passes. 
   
   In Spark parquet writer, it actually throws an exception:
   
   `java.lang.UnsupportedOperationException: Spark does not support timestamp without time zone field`
   
   If I understand correctly, it may not be fixed by using `TypeWithSchemaVisitor ` only. I will give it a try. I have made changes to throw similar exception as Spark does and comment out some tests. That was done before I found this opened issue.
   
   If it is not an easy fix, I will submit the exception PR.
   
   I also did a quick test on UUID and it also fails. I haven't debug that issue yet. I suspect similar issue of Logical Type support. 
   
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org