You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "tcguanshuhuai (via GitHub)" <gi...@apache.org> on 2024/04/08 07:08:05 UTC

[I] Iceberg use PARTITIONED BY (days(ts)) cause wrong partition name [iceberg]

tcguanshuhuai opened a new issue, #10102:
URL: https://github.com/apache/iceberg/issues/10102

   ### Apache Iceberg version
   
   1.5.0 (latest release)
   
   ### Query engine
   
   Spark
   
   ### Please describe the bug šŸž
   
   Step1: start iceberg
   spark-sql --jars /root/myjars/iceberg-spark-runtime-3.3_2.12-1.5.0.jar \
       --conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions \
       --conf spark.sql.catalog.spark_catalog=org.apache.iceberg.spark.SparkSessionCatalog \
       --conf spark.sql.catalog.spark_catalog.type=hive \
       --conf spark.sql.catalog.hive_prod=org.apache.iceberg.spark.SparkCatalog \
       --conf spark.sql.catalog.hive_prod.type=hive \
       --conf spark.sql.catalog.hadoop_prod=org.apache.iceberg.spark.SparkCatalog \
       --conf spark.sql.catalog.hadoop_prod.type=hadoop \
       --conf spark.sql.catalog.hadoop_prod.warehouse=hdfs://ns:8020/user/iceberg/warehouse/hadoop_prod
   
   Step2: create a iceberg table 
   CREATE TABLE hadoop_prod.iceberg_db.sample_hidden_partition (
       id bigint,
       data string,
       ts timestamp)
   USING iceberg
   PARTITIONED BY (days(ts));
   
   Step3: insert into a record
   //Timestamp: 1718899200  is  2024-06-21 00:00:00
   insert into table hadoop_prod.iceberg_db.sample_hidden_partition values(2, '2021-01-02', cast(1718899200 as timestamp) );
   
   We can find use select sql I can get the correct value:
   spark-sql (default)> select * from hadoop_prod.iceberg_db.sample_hidden_partition;
   id	data	ts
   2	2021-01-02	2024-06-21 00:00:00
   
   However, the hadoop url is wrong:
   spark-sql (default)> dfs -ls /user/iceberg/warehouse/hadoop_prod/iceberg_db/sample_hidden_partition/data;
   
   **Expect result:** 
   drwxr-xr-x   - root supergroup          0 2024-04-08 14:36 /user/iceberg/warehouse/hadoop_prod/iceberg_db/sample_hidden_partition/data/ts_day=2024-06-21
   
   **Actual result:**
   drwxr-xr-x   - root supergroup          0 2024-04-08 14:36 /user/iceberg/warehouse/hadoop_prod/iceberg_db/sample_hidden_partition/data/ts_day=2024-06-20
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] Iceberg use PARTITIONED BY (days(ts)) cause wrong partition name [iceberg]

Posted by "Kapkiai (via GitHub)" <gi...@apache.org>.
Kapkiai commented on issue #10102:
URL: https://github.com/apache/iceberg/issues/10102#issuecomment-2042569175

   I don’t think it’s a bug. Converting the epoch gives ā€˜2024-06-20 16:00:00’ at UTC. I believe spark inteprets timestamps at UTC based on it’s tz/system settings and you are in different timezone. I have faced similar challenges before. When I really don’t need timezone info, I mostly prefer using spark’s timestamp_ntz.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] Iceberg use PARTITIONED BY (days(ts)) cause wrong partition name [iceberg]

Posted by "tcguanshuhuai (via GitHub)" <gi...@apache.org>.
tcguanshuhuai commented on issue #10102:
URL: https://github.com/apache/iceberg/issues/10102#issuecomment-2042571758

   Thanks a lot.
   
   
   
   Sent&nbsp;from&nbsp;my&nbsp;iPhone
   
   
   ------------------ Original ------------------
   From: Mathew Kapkiai ***@***.***&gt;
   Date: Mon,Apr 8,2024 8:03 PM
   To: apache/iceberg ***@***.***&gt;
   Cc: tcguanshuhuai ***@***.***&gt;, Author ***@***.***&gt;
   Subject: Re: [apache/iceberg] Iceberg use PARTITIONED BY (days(ts)) causewrong partition name (Issue #10102)
   
   
   
   
   
    
   I don’t think it’s a bug. Converting the epoch gives ā€˜2024-06-20 16:00:00’ at UTC. I believe spark inteprets timestamps at UTC based on it’s tz/system settings and you are in different timezone. I have faced similar challenges before. When I really don’t need timezone info, I mostly prefer using spark’s timestamp_ntz.
    
   —
   Reply to this email directly, view it on GitHub, or unsubscribe.
   You are receiving this because you authored the thread.Message ID: ***@***.***&gt;


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] Iceberg use PARTITIONED BY (days(ts)) cause wrong partition name [iceberg]

Posted by "tcguanshuhuai (via GitHub)" <gi...@apache.org>.
tcguanshuhuai closed issue #10102: Iceberg use PARTITIONED BY (days(ts)) cause wrong partition name
URL: https://github.com/apache/iceberg/issues/10102


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] Iceberg use PARTITIONED BY (days(ts)) cause wrong partition name [iceberg]

Posted by "tcguanshuhuai (via GitHub)" <gi...@apache.org>.
tcguanshuhuai commented on issue #10102:
URL: https://github.com/apache/iceberg/issues/10102#issuecomment-2042573662

   Not a bug


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org