[GitHub] [iceberg] sshkvar edited a comment on pull request #2282: [SPARK] Spark parquet read timestamp without timezone

[GitBox <gi...@apache.org>]

sshkvar edited a comment on pull request #2282:
URL: https://github.com/apache/iceberg/pull/2282#issuecomment-853851066


   Hi, I am working on write support together with @daniil-dubin 
   @bkahloon @RussellSpitzer in current implementation we have `spark-handle-timestamp-without-timezone` flag, which can be used as read or write(after implementation) option, like 
   ```scala
   spark
   .read
   .format("iceberg")
   .option("spark-handle-timestamp-without-timezone", "true")
   .load("some.table.name")
   ```
   
   but we do not have ability to set this flag when using spark sql 
   ```scala
   spark.sql("select * from some.table.name") //<- will fail in current implementation
   ```
   
   Our proposal is move this flag to spark session property, @bkahloon @RussellSpitzer what do you think about it? 
   This solution potentially (need to check) will also allow us to add additional check in [TypeToSparkType.java](https://github.com/bkahloon/iceberg/blob/022dde6d4e56c373194a78a52e3f61df8d582026/spark/src/main/java/org/apache/iceberg/spark/TypeToSparkType.java#L112)
   Something like it 
   ```scala
   case TIMESTAMP:
           RuntimeConfig runtimeConfig = SparkSession.active().conf();
           boolean canHandleTimestampWithoutZone = SparkUtil.canHandleTimestampWithoutZone(runtimeConfig);
           Types.TimestampType timestamp = (Types.TimestampType) primitive;
           if (timestamp.shouldAdjustToUTC() || canHandleTimestampWithoutZone) {
             return TimestampType$.MODULE$;
           }
           throw new UnsupportedOperationException(SparkUtil.TIMESTAMP_WITHOUT_TIMEZONE_ERROR);
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org