You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "zzzzming95 (via GitHub)" <gi...@apache.org> on 2024/02/23 08:33:51 UTC

[I] Spark read failed when migrate hive orc table with `timestamp` column [iceberg]

zzzzming95 opened a new issue, #9784:
URL: https://github.com/apache/iceberg/issues/9784

   ### Apache Iceberg version
   
   None
   
   ### Query engine
   
   None
   
   ### Please describe the bug 🐞
   
   
   ```
   spark.sql("CREATE EXTERNAL TABLE mytable (foo timestamp) STORED AS orc LOCATION '/Users/russellspitzer/Temp/foo'")

   spark.sql("INSERT INTO mytable VALUES (now())")
   
spark.sql("CALL spark_catalog.system.migrate('test_timestamp')")

   
spark.sql("SELECT * FROM mytable")
   ```
   i see the old issue : https://github.com/apache/iceberg/issues/2245
   
   but when i using spark 3.4 and `iceberg-spark-runtime-3.4_2.12-1.4.3.jar` , i still get errors.
   
   ```
   Caused by: java.lang.IllegalArgumentException: Can not promote TIMESTAMP type to TIMESTAMP
       at org.apache.iceberg.relocated.com.google.common.base.Preconditions.checkArgument(Preconditions.java:445)
       at org.apache.iceberg.orc.ORCSchemaUtil.buildOrcProjection(ORCSchemaUtil.java:319)
       at org.apache.iceberg.orc.ORCSchemaUtil.buildOrcProjection(ORCSchemaUtil.java:284)
       at org.apache.iceberg.orc.ORCSchemaUtil.buildOrcProjection(ORCSchemaUtil.java:265)
   ```
   
   I think this is because hive and spark treat `timestamp` data type as timestamp with time zone and the orc file format is also stored as orc `timestamp` type. But in fact the hive `timestamp` data type should be stored as `timestamp_instant` in the orc file.
   
   
   Iceberg strictly follows the data type specification of orc, that is, orc `timestamp` is regarded as timestamp without time zone, and orc `timestamp_instant` is regarded as timestamp with time zone. This causes exceptions to occur.
   
   
   
   Please ask the community if there is a solution to this problem? thanks~


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] Spark read failed when migrate hive orc table with `timestamp` column [iceberg]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #9784:
URL: https://github.com/apache/iceberg/issues/9784#issuecomment-2425314041

   This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] Spark read failed when migrate hive orc table with `timestamp` column [iceberg]

Posted by "zzzzming95 (via GitHub)" <gi...@apache.org>.
zzzzming95 commented on issue #9784:
URL: https://github.com/apache/iceberg/issues/9784#issuecomment-1962271191

   @RussellSpitzer can you take a look on this issue? thanks


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] Spark read failed when migrate hive orc table with `timestamp` column [iceberg]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #9784: Spark read failed when migrate hive orc table with `timestamp` column 
URL: https://github.com/apache/iceberg/issues/9784


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] Spark read failed when migrate hive orc table with `timestamp` column [iceberg]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #9784:
URL: https://github.com/apache/iceberg/issues/9784#issuecomment-2455962819

   This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org