You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "tcguanshuhuai (via GitHub)" <gi...@apache.org> on 2024/04/08 07:08:05 UTC
[I] Iceberg use PARTITIONED BY (days(ts)) cause wrong partition name [iceberg]
tcguanshuhuai opened a new issue, #10102:
URL: https://github.com/apache/iceberg/issues/10102
### Apache Iceberg version
1.5.0 (latest release)
### Query engine
Spark
### Please describe the bug š
Step1: start iceberg
spark-sql --jars /root/myjars/iceberg-spark-runtime-3.3_2.12-1.5.0.jar \
--conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions \
--conf spark.sql.catalog.spark_catalog=org.apache.iceberg.spark.SparkSessionCatalog \
--conf spark.sql.catalog.spark_catalog.type=hive \
--conf spark.sql.catalog.hive_prod=org.apache.iceberg.spark.SparkCatalog \
--conf spark.sql.catalog.hive_prod.type=hive \
--conf spark.sql.catalog.hadoop_prod=org.apache.iceberg.spark.SparkCatalog \
--conf spark.sql.catalog.hadoop_prod.type=hadoop \
--conf spark.sql.catalog.hadoop_prod.warehouse=hdfs://ns:8020/user/iceberg/warehouse/hadoop_prod
Step2: create a iceberg table
CREATE TABLE hadoop_prod.iceberg_db.sample_hidden_partition (
id bigint,
data string,
ts timestamp)
USING iceberg
PARTITIONED BY (days(ts));
Step3: insert into a record
//Timestamp: 1718899200 is 2024-06-21 00:00:00
insert into table hadoop_prod.iceberg_db.sample_hidden_partition values(2, '2021-01-02', cast(1718899200 as timestamp) );
We can find use select sql I can get the correct value:
spark-sql (default)> select * from hadoop_prod.iceberg_db.sample_hidden_partition;
id data ts
2 2021-01-02 2024-06-21 00:00:00
However, the hadoop url is wrong:
spark-sql (default)> dfs -ls /user/iceberg/warehouse/hadoop_prod/iceberg_db/sample_hidden_partition/data;
**Expect result:**
drwxr-xr-x - root supergroup 0 2024-04-08 14:36 /user/iceberg/warehouse/hadoop_prod/iceberg_db/sample_hidden_partition/data/ts_day=2024-06-21
**Actual result:**
drwxr-xr-x - root supergroup 0 2024-04-08 14:36 /user/iceberg/warehouse/hadoop_prod/iceberg_db/sample_hidden_partition/data/ts_day=2024-06-20
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Iceberg use PARTITIONED BY (days(ts)) cause wrong partition name [iceberg]
Posted by "Kapkiai (via GitHub)" <gi...@apache.org>.
Kapkiai commented on issue #10102:
URL: https://github.com/apache/iceberg/issues/10102#issuecomment-2042569175
I donāt think itās a bug. Converting the epoch gives ā2024-06-20 16:00:00ā at UTC. I believe spark inteprets timestamps at UTC based on itās tz/system settings and you are in different timezone. I have faced similar challenges before. When I really donāt need timezone info, I mostly prefer using sparkās timestamp_ntz.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Iceberg use PARTITIONED BY (days(ts)) cause wrong partition name [iceberg]
Posted by "tcguanshuhuai (via GitHub)" <gi...@apache.org>.
tcguanshuhuai commented on issue #10102:
URL: https://github.com/apache/iceberg/issues/10102#issuecomment-2042571758
Thanks a lot.
Sent from my iPhone
------------------ Original ------------------
From: Mathew Kapkiai ***@***.***>
Date: Mon,Apr 8,2024 8:03 PM
To: apache/iceberg ***@***.***>
Cc: tcguanshuhuai ***@***.***>, Author ***@***.***>
Subject: Re: [apache/iceberg] Iceberg use PARTITIONED BY (days(ts)) causewrong partition name (Issue #10102)
I donāt think itās a bug. Converting the epoch gives ā2024-06-20 16:00:00ā at UTC. I believe spark inteprets timestamps at UTC based on itās tz/system settings and you are in different timezone. I have faced similar challenges before. When I really donāt need timezone info, I mostly prefer using sparkās timestamp_ntz.
ā
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you authored the thread.Message ID: ***@***.***>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Iceberg use PARTITIONED BY (days(ts)) cause wrong partition name [iceberg]
Posted by "tcguanshuhuai (via GitHub)" <gi...@apache.org>.
tcguanshuhuai closed issue #10102: Iceberg use PARTITIONED BY (days(ts)) cause wrong partition name
URL: https://github.com/apache/iceberg/issues/10102
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Iceberg use PARTITIONED BY (days(ts)) cause wrong partition name [iceberg]
Posted by "tcguanshuhuai (via GitHub)" <gi...@apache.org>.
tcguanshuhuai commented on issue #10102:
URL: https://github.com/apache/iceberg/issues/10102#issuecomment-2042573662
Not a bug
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org