You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "jamespfaulkner (via GitHub)" <gi...@apache.org> on 2025/10/22 16:36:39 UTC
[PR] Core: Avro logical timestamp-millis support [iceberg]
jamespfaulkner opened a new pull request, #14401:
URL: https://github.com/apache/iceberg/pull/14401
Hey folks,
This PR addresses the issue https://github.com/apache/iceberg/issues/12395
I've contributed a slightly modified version of the original [PR](https://github.com/apache/iceberg/pull/12397/files) addressing this issue.
The differences are:
* This includes changes to `PlannedDataReader` to account for the deprecation of `DataReader`
* `timestamp-millis` is handled independently of `timestamp-micros` to account for data files written outside of iceberg.
I tested this locally on a small avro based hive table that was migrated to iceberg and queried via a `IcebergGenerics:read` scan:
* Using the existing main branch, it recreates the `IllegalArgumentException: Unknown logical type: org.apache.hive.iceberg.org.apache.avro.LogicalTypes$TimestampMillis` issue
* Using the [PR](https://github.com/apache/iceberg/pull/12397/files) changes resulted in that Exception no longer being thrown, but the wrong DateTime was being returned.
* Both issues appeared to be fixed with these changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Avro logical timestamp-millis support [iceberg]
Posted by "jamespfaulkner (via GitHub)" <gi...@apache.org>.
jamespfaulkner commented on PR #14401:
URL: https://github.com/apache/iceberg/pull/14401#issuecomment-3498679989
Thanks @nastra and @huaxingao.
I've updated to include pre epoch test cases.
> and a test that exercises the new writer code paths
I've removed that writer now. We decided it [wasn't needed](https://github.com/apache/iceberg/pull/14401#discussion_r2494028380) but I'd forgotten to remove it previously.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Avro logical timestamp-millis support [iceberg]
Posted by "huaxingao (via GitHub)" <gi...@apache.org>.
huaxingao commented on PR #14401:
URL: https://github.com/apache/iceberg/pull/14401#issuecomment-3494570153
Thanks @jamespfaulkner for the PR! It looks good to me overall. Could we also add a test for a pre-epoch timestamp case (e.g., 1965-01-01 10:11:12.123) and a test that exercises the new writer code paths? Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Avro logical timestamp-millis support [iceberg]
Posted by "jamespfaulkner (via GitHub)" <gi...@apache.org>.
jamespfaulkner commented on code in PR #14401:
URL: https://github.com/apache/iceberg/pull/14401#discussion_r2500196305
##########
core/src/main/java/org/apache/iceberg/data/avro/GenericWriters.java:
##########
@@ -60,6 +60,14 @@ static ValueWriter<OffsetDateTime> timestamptzNanos() {
return TimestamptzNanoWriter.INSTANCE;
}
+ static ValueWriter<LocalDateTime> timestampMillis() {
Review Comment:
Done 👍
##########
core/src/test/java/org/apache/iceberg/data/avro/TestDataReader.java:
##########
@@ -0,0 +1,164 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data.avro;
+
+import static org.apache.iceberg.avro.AvroSchemaUtil.ADJUST_TO_UTC_PROP;
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.time.LocalDateTime;
+import java.time.OffsetDateTime;
+import java.time.ZoneOffset;
+import org.apache.avro.LogicalTypes;
+import org.apache.avro.Schema;
+import org.apache.avro.SchemaBuilder;
+import org.apache.avro.generic.GenericData;
+import org.apache.avro.generic.GenericDatumWriter;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.io.BinaryEncoder;
+import org.apache.avro.io.DecoderFactory;
+import org.apache.avro.io.EncoderFactory;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.DateTimeUtil;
+import org.junit.jupiter.api.Test;
+
+class TestDataReader {
+
+ @Test
+ public void testTimestampDataReader() throws IOException {
Review Comment:
Done 👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Avro logical timestamp-millis support [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra merged PR #14401:
URL: https://github.com/apache/iceberg/pull/14401
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Avro logical timestamp-millis support [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #14401:
URL: https://github.com/apache/iceberg/pull/14401#discussion_r2498595487
##########
core/src/test/java/org/apache/iceberg/data/avro/TestDataReader.java:
##########
@@ -0,0 +1,164 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data.avro;
+
+import static org.apache.iceberg.avro.AvroSchemaUtil.ADJUST_TO_UTC_PROP;
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.time.LocalDateTime;
+import java.time.OffsetDateTime;
+import java.time.ZoneOffset;
+import org.apache.avro.LogicalTypes;
+import org.apache.avro.Schema;
+import org.apache.avro.SchemaBuilder;
+import org.apache.avro.generic.GenericData;
+import org.apache.avro.generic.GenericDatumWriter;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.io.BinaryEncoder;
+import org.apache.avro.io.DecoderFactory;
+import org.apache.avro.io.EncoderFactory;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.DateTimeUtil;
+import org.junit.jupiter.api.Test;
+
+class TestDataReader {
+
+ @Test
+ public void testTimestampDataReader() throws IOException {
Review Comment:
nit: please remove all of the `test` prefixes in the newly added tests, since those aren't really adding any value
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Avro logical timestamp-millis support [iceberg]
Posted by "jamespfaulkner (via GitHub)" <gi...@apache.org>.
jamespfaulkner commented on PR #14401:
URL: https://github.com/apache/iceberg/pull/14401#issuecomment-3501563905
Thanks @huaxingao and @nastra . I appreciate the time spent reviewing!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Avro logical timestamp-millis support [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #14401:
URL: https://github.com/apache/iceberg/pull/14401#discussion_r2494033809
##########
core/src/main/java/org/apache/iceberg/data/avro/DataReader.java:
##########
@@ -146,6 +146,12 @@ public ValueReader<?> primitive(Type.PrimitiveType ignored, Schema primitive) {
}
return GenericReaders.timestampNanos();
+ case "timestamp-millis":
Review Comment:
we also need a test for this code path
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Avro logical timestamp-millis support [iceberg]
Posted by "jamespfaulkner (via GitHub)" <gi...@apache.org>.
jamespfaulkner commented on code in PR #14401:
URL: https://github.com/apache/iceberg/pull/14401#discussion_r2495485322
##########
core/src/main/java/org/apache/iceberg/data/avro/GenericReaders.java:
##########
@@ -137,6 +145,28 @@ public OffsetDateTime read(Decoder decoder, Object reuse) throws IOException {
}
}
+ private static class TimestampMilliReader implements ValueReader<LocalDateTime> {
Review Comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Avro logical timestamp-millis support [iceberg]
Posted by "jamespfaulkner (via GitHub)" <gi...@apache.org>.
jamespfaulkner commented on PR #14401:
URL: https://github.com/apache/iceberg/pull/14401#issuecomment-3492478734
@nastra Thanks for the feedback/followup review 👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Avro logical timestamp-millis support [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #14401:
URL: https://github.com/apache/iceberg/pull/14401#discussion_r2473786258
##########
core/src/main/java/org/apache/iceberg/data/avro/DataWriter.java:
##########
@@ -130,6 +130,12 @@ public ValueWriter<?> primitive(Schema primitive) {
}
return GenericWriters.timestampNanos();
+ case "timestamp-millis":
Review Comment:
there should be a test that exercises this path
##########
api/src/main/java/org/apache/iceberg/util/DateTimeUtil.java:
##########
@@ -74,6 +74,10 @@ public static LocalDateTime timestampFromNanos(long nanosFromEpoch) {
return ChronoUnit.NANOS.addTo(EPOCH, nanosFromEpoch).toLocalDateTime();
}
+ public static LocalDateTime timestampFromMillis(long millisFromEpoch) {
Review Comment:
please add tests for anything that's new to `TestDateTimeUtil`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Avro logical timestamp-millis support [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #14401:
URL: https://github.com/apache/iceberg/pull/14401#discussion_r2494028380
##########
core/src/main/java/org/apache/iceberg/data/avro/DataWriter.java:
##########
@@ -130,6 +130,12 @@ public ValueWriter<?> primitive(Schema primitive) {
}
return GenericWriters.timestampNanos();
+ case "timestamp-millis":
Review Comment:
yes indeed we should be removing "timestamp-millis" from the writers. Adding the possibility to read timestamp millis is fine, but we don't want to write it from Iceberg
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Avro logical timestamp-millis support [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #14401:
URL: https://github.com/apache/iceberg/pull/14401#discussion_r2498592118
##########
core/src/main/java/org/apache/iceberg/data/avro/GenericWriters.java:
##########
@@ -60,6 +60,14 @@ static ValueWriter<OffsetDateTime> timestamptzNanos() {
return TimestamptzNanoWriter.INSTANCE;
}
+ static ValueWriter<LocalDateTime> timestampMillis() {
Review Comment:
changes to this file can be reverted, since this isn't used anywhere
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Avro logical timestamp-millis support [iceberg]
Posted by "jamespfaulkner (via GitHub)" <gi...@apache.org>.
jamespfaulkner commented on PR #14401:
URL: https://github.com/apache/iceberg/pull/14401#issuecomment-3481763582
@nastra Thanks for the review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Avro logical timestamp-millis support [iceberg]
Posted by "jamespfaulkner (via GitHub)" <gi...@apache.org>.
jamespfaulkner commented on code in PR #14401:
URL: https://github.com/apache/iceberg/pull/14401#discussion_r2487355052
##########
core/src/main/java/org/apache/iceberg/data/avro/DataWriter.java:
##########
@@ -130,6 +130,12 @@ public ValueWriter<?> primitive(Schema primitive) {
}
return GenericWriters.timestampNanos();
+ case "timestamp-millis":
Review Comment:
So I created a test within `TestAvroDataWriter` that exercised both of the code paths under `"timestamp-millis"`. However, upon further investigation, the files written through Iceberg are either going to use the logical type `timestamp-micros` or `timestamp-nanos` (corresponding to [TimestampType](https://github.com/jamespfaulkner/iceberg/blob/e1e0a7404740b2bf9e6638afb5f0ff19f2536713/api/src/main/java/org/apache/iceberg/types/Types.java#L247) and [TimestampNanoType](https://github.com/jamespfaulkner/iceberg/blob/e1e0a7404740b2bf9e6638afb5f0ff19f2536713/api/src/main/java/org/apache/iceberg/types/Types.java#L301). I dont think we want to introduce another type.
I'm therefore thinking this change isn't required in the `DataWriter`. It would introduce inconsistencies between `avro.DataWriter` and `avro.DataReader`, but this PR is really only required to address issues with avro data written outside of iceberg.
I'm therefore thinking it's better to remove the changes from `DataWriter`.
WDYT?
##########
api/src/main/java/org/apache/iceberg/util/DateTimeUtil.java:
##########
@@ -74,6 +74,10 @@ public static LocalDateTime timestampFromNanos(long nanosFromEpoch) {
return ChronoUnit.NANOS.addTo(EPOCH, nanosFromEpoch).toLocalDateTime();
}
+ public static LocalDateTime timestampFromMillis(long millisFromEpoch) {
Review Comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Avro logical timestamp-millis support [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #14401:
URL: https://github.com/apache/iceberg/pull/14401#discussion_r2494009745
##########
core/src/main/java/org/apache/iceberg/data/avro/GenericReaders.java:
##########
@@ -137,6 +145,28 @@ public OffsetDateTime read(Decoder decoder, Object reuse) throws IOException {
}
}
+ private static class TimestampMilliReader implements ValueReader<LocalDateTime> {
Review Comment:
```suggestion
private static class TimestampMillisReader implements ValueReader<LocalDateTime> {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Avro logical timestamp-millis support [iceberg]
Posted by "jamespfaulkner (via GitHub)" <gi...@apache.org>.
jamespfaulkner commented on code in PR #14401:
URL: https://github.com/apache/iceberg/pull/14401#discussion_r2495492149
##########
core/src/main/java/org/apache/iceberg/data/avro/DataReader.java:
##########
@@ -146,6 +146,12 @@ public ValueReader<?> primitive(Type.PrimitiveType ignored, Schema primitive) {
}
return GenericReaders.timestampNanos();
+ case "timestamp-millis":
Review Comment:
Added the same tests we had for PlannedDataReader. Also added tests for the TZ code path too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org