You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "romandenysov-akv (via GitHub)" <gi...@apache.org> on 2026/10/14 14:59:32 UTC
[PR] Add partition transform fixtures [iceberg-verification]
romandenysov-akv opened a new pull request, #17:
URL: https://github.com/apache/iceberg-verification/pull/17
(no comment)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Add partition transform fixtures [iceberg-verification]
Posted by "laskoviymishka (via GitHub) via issues" <is...@iceberg.apache.org>.
laskoviymishka commented on code in PR #17:
URL: https://github.com/apache/iceberg-verification/pull/17#discussion_r4472635274
##########
table-spec/partition-transforms/bucket/cases.json:
##########
@@ -0,0 +1,148 @@
+{
+ "cases": [
+ {
+ "id": "bucket-int-100",
+ "valid": true,
+ "input": { "transform": "bucket[100]", "source_type": "int", "values": [null, 34] },
+ "decoded": { "result_types": ["int"], "values": [null, 79] },
+ "clause": "Bucket Transform Details and Appendix B: int hash reference vector",
+ "spec_ref": "format/spec.md#appendix-b-32-bit-hash-requirements"
+ },
+ {
+ "id": "bucket-long-100",
+ "valid": true,
+ "input": { "transform": "bucket[100]", "source_type": "long", "values": [null, "34"] },
+ "decoded": { "result_types": ["int"], "values": [null, 79] },
+ "clause": "Appendix B: integer and long hash results must be identical for all integer values",
+ "spec_ref": "format/spec.md#appendix-b-32-bit-hash-requirements"
+ },
+ {
+ "id": "bucket-decimal-9-2-100",
+ "valid": true,
+ "input": { "transform": "bucket[100]", "source_type": "decimal(9, 2)", "values": [null, "14.20"] },
+ "decoded": { "result_types": ["int"], "values": [null, 59] },
+ "clause": "Appendix B: decimal 14.20 hashes from its minimum two's-complement big-endian unscaled value; bucket discards the sign bit",
+ "spec_ref": "format/spec.md#appendix-b-32-bit-hash-requirements"
+ },
+ {
+ "id": "bucket-decimal-10-3-100",
+ "valid": true,
+ "input": { "transform": "bucket[100]", "source_type": "decimal(10, 3)", "values": [null, "1.420"] },
+ "decoded": { "result_types": ["int"], "values": [null, 59] },
+ "clause": "Appendix B: decimal hashes use the unscaled value and hash results are not dependent on decimal scale",
+ "spec_ref": "format/spec.md#appendix-b-32-bit-hash-requirements"
+ },
+ {
+ "id": "bucket-date-100",
+ "valid": true,
+ "input": { "transform": "bucket[100]", "source_type": "date", "values": [null, "2017-11-16"] },
+ "decoded": { "result_types": ["int"], "values": [null, 26] },
+ "clause": "Appendix B: date hash reference vector; bucket discards the sign bit",
+ "spec_ref": "format/spec.md#appendix-b-32-bit-hash-requirements"
+ },
+ {
+ "id": "bucket-time-100",
+ "valid": true,
+ "input": { "transform": "bucket[100]", "source_type": "time", "values": [null, "22:31:08"] },
+ "decoded": { "result_types": ["int"], "values": [null, 59] },
+ "clause": "Appendix B: time hash reference vector; bucket discards the sign bit",
+ "spec_ref": "format/spec.md#appendix-b-32-bit-hash-requirements"
+ },
+ {
+ "id": "bucket-timestamp-100",
+ "valid": true,
+ "input": {
+ "transform": "bucket[100]",
+ "source_type": "timestamp",
+ "values": [null, "2017-11-16T22:31:08", "2017-11-16T22:31:08.000001"]
+ },
+ "decoded": { "result_types": ["int"], "values": [null, 7, 38] },
+ "clause": "Appendix B: timestamp hash reference vectors at whole-second and microsecond precision",
+ "spec_ref": "format/spec.md#appendix-b-32-bit-hash-requirements"
+ },
+ {
+ "id": "bucket-timestamptz-100",
+ "valid": true,
+ "input": {
+ "transform": "bucket[100]",
+ "source_type": "timestamptz",
+ "values": [
+ null,
+ "2017-11-16T14:31:08-08:00",
+ "2017-11-16T22:31:08Z",
+ "2017-11-16T14:31:08.000001-08:00",
+ "2017-11-16T22:31:08.000001Z"
+ ]
+ },
+ "decoded": { "result_types": ["int"], "values": [null, 7, 7, 38, 38] },
+ "clause": "Appendix B: timestamptz hashes microseconds from the Unix epoch, so equivalent instants hash identically",
+ "spec_ref": "format/spec.md#appendix-b-32-bit-hash-requirements"
+ },
+ {
+ "id": "bucket-timestamp-ns-100",
+ "valid": true,
+ "input": {
+ "transform": "bucket[100]",
+ "source_type": "timestamp_ns",
+ "values": [null, "2017-11-16T22:31:08", "2017-11-16T22:31:08.000001001", "2017-11-16T22:31:08.000001999"]
+ },
+ "decoded": { "result_types": ["int"], "values": [null, 7, 38, 38] },
Review Comment:
Could we add a pre-epoch value to pin which way the ns-to-µs conversion rounds? Every ns input here is a 2017 instant, and for those floor and truncation give the same bucket. With a Murmur3 that reproduces the Appendix B vectors, `1969-12-31T23:59:59.999999999` lands in bucket 12 under floor (-1 µs) and 76 under truncation (0 µs). The spec only says ["converted to microsecond precision"](https://github.com/apache/iceberg/blob/ea73769d120dc3819d8cfa2253b6dba4bb394886/format/spec.md?plain=1#L2019) with no direction, so the 12 below follows Java's [`nanosToMicros`](https://github.com/apache/iceberg/blob/ea73769d120dc3819d8cfa2253b6dba4bb394886/api/src/main/java/org/apache/iceberg/util/DateTimeUtil.java#L104-L106) (`Math.floorDiv`), not an explicit rule in the standard. The same value with `Z` belongs in `bucket-timestamptz-ns-100`.
```suggestion
"values": [
null,
"2017-11-16T22:31:08",
"2017-11-16T22:31:08.000001001",
"2017-11-16T22:31:08.000001999",
"1969-12-31T23:59:59.999999999"
]
},
"decoded": { "result_types": ["int"], "values": [null, 7, 38, 38, 12] },
```
##########
table-spec/partition-transforms/README.md:
##########
@@ -0,0 +1,165 @@
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+# Partition transforms
+
+Binding a partition transform to a source type and applying it to values
+produces the same logical values and result type in every implementation. The
+Iceberg specification is the source of truth for this surface.
+
+## Assertion
+
+```
+apply(bind(transform, source_type), ordered_inputs) == ordered_expected_values
+result_type(bind(transform, source_type)) is one of allowed_result_types
+```
+
+Bytes are not compared. Each implementation decodes the language-neutral input
+values, binds the transform, applies it in order, and maps its results back to
+the language-neutral representation below.
+
+## Scope
+
+This surface covers the value transform and its result type. It binds every
+standard transform to each source type permitted by the Partition Transforms
+table, subject to the deliberate omissions below. The cases include the spec's
+examples, boundary values, and an explicit `null` input for every
+transform/source-type path. As required by the spec, "All transforms must return
+`null` for a `null` input value."
+
+Consumers subscribe by subdirectory:
+
+- `identity/` - "Source value, unmodified."
+- `bucket/` - Murmur3 x86 32-bit, seed 0, with the sign bit discarded before
+ taking modulo `N`.
+- `truncate/` - integer, long, decimal, string, and binary truncation.
+- `year/`, `month/`, `day/`, and `hour/` - temporal ordinals from the Unix
+ epoch. `day` expected values are logical dates.
+- `void/` - "Always produces `null`."
+
+The subdirectories are independent subscription boundaries. For example, an
+implementation without nanosecond timestamp types may skip cases for those
+source types while running the other cases in the same directory.
+
+## Case format
+
+Each `cases.json` holds a `cases` array. A case has this shape:
+
+```json
+{
+ "id": "bucket-int-100",
+ "valid": true,
+ "input": {
+ "transform": "bucket[100]",
+ "source_type": "int",
+ "values": [34, null]
+ },
+ "decoded": {
+ "result_types": ["int"],
+ "values": [79, null]
Review Comment:
Could the example match the real `bucket-int-100` case? It reuses that id but puts `null` last, while `bucket/cases.json` has `[null, 34]` and `[null, 79]`, like every other case in the surface.
```suggestion
"values": [null, 34]
},
"decoded": {
"result_types": ["int"],
"values": [null, 79]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Add partition transform fixtures [iceberg-verification]
Posted by "romandenysov-akv (via GitHub) via issues" <is...@iceberg.apache.org>.
romandenysov-akv commented on PR #17:
URL: https://github.com/apache/iceberg-verification/pull/17#issuecomment-6771710559
> @romandenysov-akv add a small script under dev/ that recomputes the expected values and ready to merge
@laskoviymishka Resolved all comments
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Add partition transform fixtures [iceberg-verification]
Posted by "laskoviymishka (via GitHub) via issues" <is...@iceberg.apache.org>.
laskoviymishka commented on PR #17:
URL: https://github.com/apache/iceberg-verification/pull/17#issuecomment-6753033852
@romandenysov-akv add a small script under dev/ that recomputes the expected values and ready to merge
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Add partition transform fixtures [iceberg-verification]
Posted by "sungwy (via GitHub)" <gi...@apache.org>.
sungwy commented on PR #17:
URL: https://github.com/apache/iceberg-verification/pull/17#issuecomment-6371104944
Thanks @romandenysov-akv - just coming back from some work travels. I'll review this shortly
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Add partition transform fixtures [iceberg-verification]
Posted by "romandenysov-akv (via GitHub) via issues" <is...@iceberg.apache.org>.
romandenysov-akv commented on code in PR #17:
URL: https://github.com/apache/iceberg-verification/pull/17#discussion_r4481931724
##########
table-spec/partition-transforms/bucket/cases.json:
##########
@@ -0,0 +1,148 @@
+{
+ "cases": [
+ {
+ "id": "bucket-int-100",
+ "valid": true,
+ "input": { "transform": "bucket[100]", "source_type": "int", "values": [null, 34] },
+ "decoded": { "result_types": ["int"], "values": [null, 79] },
+ "clause": "Bucket Transform Details and Appendix B: int hash reference vector",
+ "spec_ref": "format/spec.md#appendix-b-32-bit-hash-requirements"
+ },
+ {
+ "id": "bucket-long-100",
+ "valid": true,
+ "input": { "transform": "bucket[100]", "source_type": "long", "values": [null, "34"] },
+ "decoded": { "result_types": ["int"], "values": [null, 79] },
+ "clause": "Appendix B: integer and long hash results must be identical for all integer values",
+ "spec_ref": "format/spec.md#appendix-b-32-bit-hash-requirements"
+ },
+ {
+ "id": "bucket-decimal-9-2-100",
+ "valid": true,
+ "input": { "transform": "bucket[100]", "source_type": "decimal(9, 2)", "values": [null, "14.20"] },
+ "decoded": { "result_types": ["int"], "values": [null, 59] },
+ "clause": "Appendix B: decimal 14.20 hashes from its minimum two's-complement big-endian unscaled value; bucket discards the sign bit",
+ "spec_ref": "format/spec.md#appendix-b-32-bit-hash-requirements"
+ },
+ {
+ "id": "bucket-decimal-10-3-100",
+ "valid": true,
+ "input": { "transform": "bucket[100]", "source_type": "decimal(10, 3)", "values": [null, "1.420"] },
+ "decoded": { "result_types": ["int"], "values": [null, 59] },
+ "clause": "Appendix B: decimal hashes use the unscaled value and hash results are not dependent on decimal scale",
+ "spec_ref": "format/spec.md#appendix-b-32-bit-hash-requirements"
+ },
+ {
+ "id": "bucket-date-100",
+ "valid": true,
+ "input": { "transform": "bucket[100]", "source_type": "date", "values": [null, "2017-11-16"] },
+ "decoded": { "result_types": ["int"], "values": [null, 26] },
+ "clause": "Appendix B: date hash reference vector; bucket discards the sign bit",
+ "spec_ref": "format/spec.md#appendix-b-32-bit-hash-requirements"
+ },
+ {
+ "id": "bucket-time-100",
+ "valid": true,
+ "input": { "transform": "bucket[100]", "source_type": "time", "values": [null, "22:31:08"] },
+ "decoded": { "result_types": ["int"], "values": [null, 59] },
+ "clause": "Appendix B: time hash reference vector; bucket discards the sign bit",
+ "spec_ref": "format/spec.md#appendix-b-32-bit-hash-requirements"
+ },
+ {
+ "id": "bucket-timestamp-100",
+ "valid": true,
+ "input": {
+ "transform": "bucket[100]",
+ "source_type": "timestamp",
+ "values": [null, "2017-11-16T22:31:08", "2017-11-16T22:31:08.000001"]
+ },
+ "decoded": { "result_types": ["int"], "values": [null, 7, 38] },
+ "clause": "Appendix B: timestamp hash reference vectors at whole-second and microsecond precision",
+ "spec_ref": "format/spec.md#appendix-b-32-bit-hash-requirements"
+ },
+ {
+ "id": "bucket-timestamptz-100",
+ "valid": true,
+ "input": {
+ "transform": "bucket[100]",
+ "source_type": "timestamptz",
+ "values": [
+ null,
+ "2017-11-16T14:31:08-08:00",
+ "2017-11-16T22:31:08Z",
+ "2017-11-16T14:31:08.000001-08:00",
+ "2017-11-16T22:31:08.000001Z"
+ ]
+ },
+ "decoded": { "result_types": ["int"], "values": [null, 7, 7, 38, 38] },
+ "clause": "Appendix B: timestamptz hashes microseconds from the Unix epoch, so equivalent instants hash identically",
+ "spec_ref": "format/spec.md#appendix-b-32-bit-hash-requirements"
+ },
+ {
+ "id": "bucket-timestamp-ns-100",
+ "valid": true,
+ "input": {
+ "transform": "bucket[100]",
+ "source_type": "timestamp_ns",
+ "values": [null, "2017-11-16T22:31:08", "2017-11-16T22:31:08.000001001", "2017-11-16T22:31:08.000001999"]
+ },
+ "decoded": { "result_types": ["int"], "values": [null, 7, 38, 38] },
Review Comment:
@laskoviymishka Thanks for the example. It distinguishes floor division (bucket 12) from truncation toward zero (bucket 76).
I checked Iceberg Java, Go, and Rust; all three use floor division, and Rust explicitly changed its implementation to match Java.
Appendix B requires conversion to microseconds but does not state the rounding rule for negative values between microseconds.
Should we clarify that rule in the spec before making bucket 12 a conformance expectation, or do you think the current wording is sufficient to include this case in the PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Add partition transform fixtures [iceberg-verification]
Posted by "romandenysov-akv (via GitHub) via issues" <is...@iceberg.apache.org>.
romandenysov-akv commented on code in PR #17:
URL: https://github.com/apache/iceberg-verification/pull/17#discussion_r4515311673
##########
table-spec/partition-transforms/bucket/cases.json:
##########
@@ -0,0 +1,148 @@
+{
+ "cases": [
+ {
+ "id": "bucket-int-100",
+ "valid": true,
+ "input": { "transform": "bucket[100]", "source_type": "int", "values": [null, 34] },
+ "decoded": { "result_types": ["int"], "values": [null, 79] },
+ "clause": "Bucket Transform Details and Appendix B: int hash reference vector",
+ "spec_ref": "format/spec.md#appendix-b-32-bit-hash-requirements"
+ },
+ {
+ "id": "bucket-long-100",
+ "valid": true,
+ "input": { "transform": "bucket[100]", "source_type": "long", "values": [null, "34"] },
+ "decoded": { "result_types": ["int"], "values": [null, 79] },
+ "clause": "Appendix B: integer and long hash results must be identical for all integer values",
+ "spec_ref": "format/spec.md#appendix-b-32-bit-hash-requirements"
+ },
+ {
+ "id": "bucket-decimal-9-2-100",
+ "valid": true,
+ "input": { "transform": "bucket[100]", "source_type": "decimal(9, 2)", "values": [null, "14.20"] },
+ "decoded": { "result_types": ["int"], "values": [null, 59] },
+ "clause": "Appendix B: decimal 14.20 hashes from its minimum two's-complement big-endian unscaled value; bucket discards the sign bit",
+ "spec_ref": "format/spec.md#appendix-b-32-bit-hash-requirements"
+ },
+ {
+ "id": "bucket-decimal-10-3-100",
+ "valid": true,
+ "input": { "transform": "bucket[100]", "source_type": "decimal(10, 3)", "values": [null, "1.420"] },
+ "decoded": { "result_types": ["int"], "values": [null, 59] },
+ "clause": "Appendix B: decimal hashes use the unscaled value and hash results are not dependent on decimal scale",
+ "spec_ref": "format/spec.md#appendix-b-32-bit-hash-requirements"
+ },
+ {
+ "id": "bucket-date-100",
+ "valid": true,
+ "input": { "transform": "bucket[100]", "source_type": "date", "values": [null, "2017-11-16"] },
+ "decoded": { "result_types": ["int"], "values": [null, 26] },
+ "clause": "Appendix B: date hash reference vector; bucket discards the sign bit",
+ "spec_ref": "format/spec.md#appendix-b-32-bit-hash-requirements"
+ },
+ {
+ "id": "bucket-time-100",
+ "valid": true,
+ "input": { "transform": "bucket[100]", "source_type": "time", "values": [null, "22:31:08"] },
+ "decoded": { "result_types": ["int"], "values": [null, 59] },
+ "clause": "Appendix B: time hash reference vector; bucket discards the sign bit",
+ "spec_ref": "format/spec.md#appendix-b-32-bit-hash-requirements"
+ },
+ {
+ "id": "bucket-timestamp-100",
+ "valid": true,
+ "input": {
+ "transform": "bucket[100]",
+ "source_type": "timestamp",
+ "values": [null, "2017-11-16T22:31:08", "2017-11-16T22:31:08.000001"]
+ },
+ "decoded": { "result_types": ["int"], "values": [null, 7, 38] },
+ "clause": "Appendix B: timestamp hash reference vectors at whole-second and microsecond precision",
+ "spec_ref": "format/spec.md#appendix-b-32-bit-hash-requirements"
+ },
+ {
+ "id": "bucket-timestamptz-100",
+ "valid": true,
+ "input": {
+ "transform": "bucket[100]",
+ "source_type": "timestamptz",
+ "values": [
+ null,
+ "2017-11-16T14:31:08-08:00",
+ "2017-11-16T22:31:08Z",
+ "2017-11-16T14:31:08.000001-08:00",
+ "2017-11-16T22:31:08.000001Z"
+ ]
+ },
+ "decoded": { "result_types": ["int"], "values": [null, 7, 7, 38, 38] },
+ "clause": "Appendix B: timestamptz hashes microseconds from the Unix epoch, so equivalent instants hash identically",
+ "spec_ref": "format/spec.md#appendix-b-32-bit-hash-requirements"
+ },
+ {
+ "id": "bucket-timestamp-ns-100",
+ "valid": true,
+ "input": {
+ "transform": "bucket[100]",
+ "source_type": "timestamp_ns",
+ "values": [null, "2017-11-16T22:31:08", "2017-11-16T22:31:08.000001001", "2017-11-16T22:31:08.000001999"]
+ },
+ "decoded": { "result_types": ["int"], "values": [null, 7, 38, 38] },
Review Comment:
Opened thread:
https://lists.apache.org/thread/hcc1yo2pcyrc089rgywo8nh0tt9gs5yl
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Add partition transform fixtures [iceberg-verification]
Posted by "laskoviymishka (via GitHub) via issues" <is...@iceberg.apache.org>.
laskoviymishka commented on PR #17:
URL: https://github.com/apache/iceberg-verification/pull/17#issuecomment-6847250584
@romandenysov-akv i think this covers everything and ready to merge
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Add partition transform fixtures [iceberg-verification]
Posted by "romandenysov-akv (via GitHub)" <gi...@apache.org>.
romandenysov-akv commented on PR #17:
URL: https://github.com/apache/iceberg-verification/pull/17#issuecomment-6190488082
Hello @sungwy ,
I prepared contribution to resolve current issue
https://github.com/apache/iceberg-verification/pull/17
Could you take a look when you have a chance?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Add partition transform fixtures [iceberg-verification]
Posted by "laskoviymishka (via GitHub) via issues" <is...@iceberg.apache.org>.
laskoviymishka commented on code in PR #17:
URL: https://github.com/apache/iceberg-verification/pull/17#discussion_r4490147288
##########
table-spec/partition-transforms/bucket/cases.json:
##########
@@ -0,0 +1,148 @@
+{
+ "cases": [
+ {
+ "id": "bucket-int-100",
+ "valid": true,
+ "input": { "transform": "bucket[100]", "source_type": "int", "values": [null, 34] },
+ "decoded": { "result_types": ["int"], "values": [null, 79] },
+ "clause": "Bucket Transform Details and Appendix B: int hash reference vector",
+ "spec_ref": "format/spec.md#appendix-b-32-bit-hash-requirements"
+ },
+ {
+ "id": "bucket-long-100",
+ "valid": true,
+ "input": { "transform": "bucket[100]", "source_type": "long", "values": [null, "34"] },
+ "decoded": { "result_types": ["int"], "values": [null, 79] },
+ "clause": "Appendix B: integer and long hash results must be identical for all integer values",
+ "spec_ref": "format/spec.md#appendix-b-32-bit-hash-requirements"
+ },
+ {
+ "id": "bucket-decimal-9-2-100",
+ "valid": true,
+ "input": { "transform": "bucket[100]", "source_type": "decimal(9, 2)", "values": [null, "14.20"] },
+ "decoded": { "result_types": ["int"], "values": [null, 59] },
+ "clause": "Appendix B: decimal 14.20 hashes from its minimum two's-complement big-endian unscaled value; bucket discards the sign bit",
+ "spec_ref": "format/spec.md#appendix-b-32-bit-hash-requirements"
+ },
+ {
+ "id": "bucket-decimal-10-3-100",
+ "valid": true,
+ "input": { "transform": "bucket[100]", "source_type": "decimal(10, 3)", "values": [null, "1.420"] },
+ "decoded": { "result_types": ["int"], "values": [null, 59] },
+ "clause": "Appendix B: decimal hashes use the unscaled value and hash results are not dependent on decimal scale",
+ "spec_ref": "format/spec.md#appendix-b-32-bit-hash-requirements"
+ },
+ {
+ "id": "bucket-date-100",
+ "valid": true,
+ "input": { "transform": "bucket[100]", "source_type": "date", "values": [null, "2017-11-16"] },
+ "decoded": { "result_types": ["int"], "values": [null, 26] },
+ "clause": "Appendix B: date hash reference vector; bucket discards the sign bit",
+ "spec_ref": "format/spec.md#appendix-b-32-bit-hash-requirements"
+ },
+ {
+ "id": "bucket-time-100",
+ "valid": true,
+ "input": { "transform": "bucket[100]", "source_type": "time", "values": [null, "22:31:08"] },
+ "decoded": { "result_types": ["int"], "values": [null, 59] },
+ "clause": "Appendix B: time hash reference vector; bucket discards the sign bit",
+ "spec_ref": "format/spec.md#appendix-b-32-bit-hash-requirements"
+ },
+ {
+ "id": "bucket-timestamp-100",
+ "valid": true,
+ "input": {
+ "transform": "bucket[100]",
+ "source_type": "timestamp",
+ "values": [null, "2017-11-16T22:31:08", "2017-11-16T22:31:08.000001"]
+ },
+ "decoded": { "result_types": ["int"], "values": [null, 7, 38] },
+ "clause": "Appendix B: timestamp hash reference vectors at whole-second and microsecond precision",
+ "spec_ref": "format/spec.md#appendix-b-32-bit-hash-requirements"
+ },
+ {
+ "id": "bucket-timestamptz-100",
+ "valid": true,
+ "input": {
+ "transform": "bucket[100]",
+ "source_type": "timestamptz",
+ "values": [
+ null,
+ "2017-11-16T14:31:08-08:00",
+ "2017-11-16T22:31:08Z",
+ "2017-11-16T14:31:08.000001-08:00",
+ "2017-11-16T22:31:08.000001Z"
+ ]
+ },
+ "decoded": { "result_types": ["int"], "values": [null, 7, 7, 38, 38] },
+ "clause": "Appendix B: timestamptz hashes microseconds from the Unix epoch, so equivalent instants hash identically",
+ "spec_ref": "format/spec.md#appendix-b-32-bit-hash-requirements"
+ },
+ {
+ "id": "bucket-timestamp-ns-100",
+ "valid": true,
+ "input": {
+ "transform": "bucket[100]",
+ "source_type": "timestamp_ns",
+ "values": [null, "2017-11-16T22:31:08", "2017-11-16T22:31:08.000001001", "2017-11-16T22:31:08.000001999"]
+ },
+ "decoded": { "result_types": ["int"], "values": [null, 7, 38, 38] },
Review Comment:
Let's file a thread on @dev-list first to open a discussion about this adjustment, it seems to me important enough to have a note in spec.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Add partition transform fixtures [iceberg-verification]
Posted by "laskoviymishka (via GitHub) via issues" <is...@iceberg.apache.org>.
laskoviymishka commented on code in PR #17:
URL: https://github.com/apache/iceberg-verification/pull/17#discussion_r4536787137
##########
dev/schema/cases.partition-transforms.schema.json:
##########
@@ -0,0 +1,99 @@
+{
+ "$schema": "https://json-schema.org/draft/2020-12/schema",
+ "$id": "https://iceberg.apache.org/verification/cases.partition-transforms.schema.json",
+ "title": "Conformance cases (partition-transforms surface extension)",
+ "description": "Extra structural constraints for table-spec/partition-transforms. Applied on top of cases.base.schema.json by dev/validate-fixtures.py.",
+ "type": "object",
+ "required": ["cases"],
+ "additionalProperties": false,
+ "properties": {
+ "cases": {
+ "type": "array",
+ "items": { "$ref": "#/$defs/case" }
+ }
+ },
+ "$defs": {
+ "case": {
+ "type": "object",
+ "required": ["id", "valid", "input", "decoded", "clause", "spec_ref"],
+ "additionalProperties": false,
+ "properties": {
+ "id": { "type": "string", "minLength": 1 },
+ "valid": { "const": true },
+ "input": { "$ref": "#/$defs/input" },
+ "decoded": { "$ref": "#/$defs/decoded" },
+ "clause": { "type": "string", "minLength": 1 },
+ "spec_ref": { "type": "string", "minLength": 1 },
+ "normative_level": { "enum": ["must", "should"] }
+ }
+ },
+ "input": {
+ "type": "object",
+ "required": ["transform", "source_type", "values"],
+ "additionalProperties": false,
+ "properties": {
+ "transform": {
+ "type": "string",
+ "pattern": "^(identity|bucket\\[[0-9]+\\]|truncate\\[[0-9]+\\]|year|month|day|hour|void)$"
Review Comment:
`truncate[0]` gets through this pattern and the validator: a `truncate/` case using it validates `OK`, and so does `bucket[0]` outside `bucket/`. Java's [`Truncate`](https://github.com/apache/iceberg/blob/ea73769d120dc3819d8cfa2253b6dba4bb394886/api/src/main/java/org/apache/iceberg/transforms/Truncate.java#L41-L53) rejects width <= 0 at construction, so such a fixture would expect results from a transform that implementations refuse to build.
##########
dev/validate-fixtures.py:
##########
@@ -47,10 +52,317 @@ def _is_types_surface(path):
return "/table-spec/types/" in p or p.startswith("table-spec/types/")
+def _is_partition_transforms_surface(path):
+ p = _rel(path)
+ return "/table-spec/partition-transforms/" in p or p.startswith(
+ "table-spec/partition-transforms/"
+ )
+
+
+IDENTITY_TYPES = {
+ "unknown",
+ "boolean",
+ "int",
+ "long",
+ "float",
+ "double",
+ "decimal",
+ "date",
+ "time",
+ "timestamp",
+ "timestamptz",
+ "timestamp_ns",
+ "timestamptz_ns",
+ "string",
+ "uuid",
+ "fixed",
+ "binary",
+}
+BUCKET_TYPES = {
+ "int",
+ "long",
+ "decimal",
+ "date",
+ "time",
+ "timestamp",
+ "timestamptz",
+ "timestamp_ns",
+ "timestamptz_ns",
+ "string",
+ "uuid",
+ "fixed",
+ "binary",
+}
+TRUNCATE_TYPES = {"int", "long", "decimal", "string", "binary"}
+DATE_TRANSFORM_TYPES = {
+ "date",
+ "timestamp",
+ "timestamptz",
+ "timestamp_ns",
+ "timestamptz_ns",
+}
+HOUR_TYPES = {"timestamp", "timestamptz", "timestamp_ns", "timestamptz_ns"}
+VOID_TYPES = IDENTITY_TYPES | {"geometry", "geography"}
+
+
+def _type_family(type_name):
+ if not isinstance(type_name, str):
+ return None
+ if type_name.startswith("decimal("):
+ return "decimal"
+ if type_name.startswith("fixed["):
+ return "fixed"
+ if type_name.startswith("geometry("):
+ return "geometry"
+ if type_name.startswith("geography("):
+ return "geography"
+ return type_name
+
+
+def _transform_family(transform):
+ if not isinstance(transform, str):
+ return None
+ if transform.startswith("bucket["):
+ return "bucket"
+ if transform.startswith("truncate["):
+ return "truncate"
+ return transform
+
+
+def _json_scalar_equal(left, right):
+ return type(left) is type(right) and left == right
+
+
+def _partition_transform_errors(path, doc, check_bucket=True):
+ errors = []
+ cases = doc.get("cases") if isinstance(doc, dict) else None
+ if not isinstance(cases, list):
+ return errors
+
+ allowed_sources = {
+ "identity": IDENTITY_TYPES,
+ "bucket": BUCKET_TYPES,
+ "truncate": TRUNCATE_TYPES,
+ "year": DATE_TRANSFORM_TYPES,
+ "month": DATE_TRANSFORM_TYPES,
+ "day": DATE_TRANSFORM_TYPES,
+ "hour": HOUR_TYPES,
+ "void": VOID_TYPES,
+ }
+
+ for index, case in enumerate(cases):
+ if not isinstance(case, dict):
+ continue
+ input_value = case.get("input")
+ decoded = case.get("decoded")
+ if not isinstance(input_value, dict) or not isinstance(decoded, dict):
+ continue
+
+ transform = input_value.get("transform")
+ source_type = input_value.get("source_type")
+ values = input_value.get("values")
+ result_types = decoded.get("result_types")
+ expected = decoded.get("values")
+ transform_family = _transform_family(transform)
+ source_family = _type_family(source_type)
+ label = f"{path}: cases/{index}"
+
+ permitted = allowed_sources.get(transform_family)
+ if permitted is None:
+ errors.append(f"{label}: unknown transform '{transform}'")
+ elif source_family not in permitted:
+ errors.append(
+ f"{label}: transform '{transform}' does not support source type "
+ f"'{source_type}'"
+ )
+
+ if isinstance(values, list) and isinstance(expected, list):
+ if len(values) != len(expected):
+ errors.append(
+ f"{label}: input.values and decoded.values must have equal lengths"
+ )
+ if None not in values:
+ errors.append(f"{label}: input.values must include null")
+ for value_index, (value, output) in enumerate(zip(values, expected)):
+ if value is None and output is not None:
+ errors.append(
+ f"{label}: decoded.values/{value_index} must be null for a "
+ "null input"
+ )
+ if transform_family == "void" and output is not None:
+ errors.append(
+ f"{label}: decoded.values/{value_index} must be null for void"
+ )
+ if transform_family == "identity" and not _json_scalar_equal(
+ value, output
+ ):
+ errors.append(
+ f"{label}: decoded.values/{value_index} must equal its "
+ "identity input"
+ )
+
+ required_result_types = None
+ if transform_family in {"bucket", "year", "month", "hour"}:
+ required_result_types = {"int"}
+ elif transform_family == "day":
+ required_result_types = {"date"}
+ elif transform_family in {"identity", "truncate"} and isinstance(
+ source_type, str
+ ):
+ required_result_types = {source_type}
+ elif transform_family == "void" and isinstance(source_type, str):
+ required_result_types = {source_type, "int"}
+
+ if (
+ required_result_types is not None
+ and isinstance(result_types, list)
+ and all(isinstance(result_type, str) for result_type in result_types)
+ ):
+ if set(result_types) != required_result_types:
+ errors.append(
+ f"{label}: decoded.result_types must be exactly "
+ f"{sorted(required_result_types)}"
+ )
+
+ bucket_path = path.replace("\\", "/").endswith(
+ "/partition-transforms/bucket/cases.json"
+ )
+ if check_bucket and not errors and bucket_path:
+ errors.extend(_bucket_errors(path, doc))
+
+ return errors
+
+
+EPOCH = datetime(1970, 1, 1, tzinfo=timezone.utc)
+TIMESTAMP = re.compile(
+ r"^(\d{4}-\d\d-\d\dT\d\d:\d\d:\d\d)(?:\.(\d{1,9}))?(Z|[+-]\d\d:\d\d)?$"
+)
+MASK = 0xFFFFFFFF
+
+
+def murmur3_x86_32(data):
+ """MurmurHash3 x86 32-bit with seed 0, returned as an unsigned integer."""
+ h = 0
+ for offset in range(0, len(data) // 4 * 4, 4):
+ k = int.from_bytes(data[offset : offset + 4], "little")
+ k = (k * 0xCC9E2D51) & MASK
+ k = ((k << 15) | (k >> 17)) & MASK
+ k = (k * 0x1B873593) & MASK
+ h ^= k
+ h = ((h << 13) | (h >> 19)) & MASK
+ h = (h * 5 + 0xE6546B64) & MASK
+
+ tail = data[len(data) // 4 * 4 :]
+ if tail:
+ k = int.from_bytes(tail, "little")
+ k = (k * 0xCC9E2D51) & MASK
+ k = ((k << 15) | (k >> 17)) & MASK
+ k = (k * 0x1B873593) & MASK
+ h ^= k
+
+ h ^= len(data)
+ h ^= h >> 16
+ h = (h * 0x85EBCA6B) & MASK
+ h ^= h >> 13
+ h = (h * 0xC2B2AE35) & MASK
+ return (h ^ (h >> 16)) & MASK
+
+
+def epoch_units(value, nanos=False):
+ match = TIMESTAMP.fullmatch(value)
+ if not match:
+ raise ValueError(f"invalid timestamp: {value}")
+ base, fraction, zone = match.groups()
+ instant = datetime.fromisoformat(base + ("+00:00" if zone == "Z" else zone or ""))
+ instant = instant.astimezone(timezone.utc) if zone else instant.replace(tzinfo=timezone.utc)
+ delta = instant - EPOCH
+ seconds = delta.days * 86400 + delta.seconds
+ digits = (fraction or "").ljust(9, "0")
+ nanoseconds = seconds * 1_000_000_000 + int(digits or "0")
+ if nanos and nanoseconds < 0 and nanoseconds % 1000:
+ raise ValueError("negative nanoseconds between microseconds need a spec rounding rule")
+ return nanoseconds // 1000
+
+
+def decimal_bytes(value):
+ decimal = Decimal(value)
+ digits = decimal.as_tuple()
+ unscaled = int("".join(map(str, digits.digits)))
+ if digits.sign:
+ unscaled = -unscaled
+ length = 1
+ while not -(1 << (length * 8 - 1)) <= unscaled < (1 << (length * 8 - 1)):
+ length += 1
+ return unscaled.to_bytes(length, "big", signed=True)
+
+
+def encode_for_hash(source_type, value):
+ if source_type in ("int", "long"):
+ return struct.pack("<q", int(value))
Review Comment:
For `int`, `long` and decimal the bucket check hashes whatever it's given, so values that break the [README encoding](https://github.com/apache/iceberg-verification/blob/a45307a7e7fa019fc339d03ba317387038030b12/table-spec/partition-transforms/README.md?plain=1#L100-L104) are accepted: `int` `34.5` (hashed as 34), `true` (hashed as 1), a bare-number `long` `34`, and `"14.2"` under `decimal(9, 2)`, since `decimal_bytes` never reads `P` or `S`. The committed fixtures all follow the encoding.
##########
dev/validate-fixtures.py:
##########
@@ -47,10 +52,317 @@ def _is_types_surface(path):
return "/table-spec/types/" in p or p.startswith("table-spec/types/")
+def _is_partition_transforms_surface(path):
+ p = _rel(path)
+ return "/table-spec/partition-transforms/" in p or p.startswith(
+ "table-spec/partition-transforms/"
+ )
+
+
+IDENTITY_TYPES = {
+ "unknown",
+ "boolean",
+ "int",
+ "long",
+ "float",
+ "double",
+ "decimal",
+ "date",
+ "time",
+ "timestamp",
+ "timestamptz",
+ "timestamp_ns",
+ "timestamptz_ns",
+ "string",
+ "uuid",
+ "fixed",
+ "binary",
+}
+BUCKET_TYPES = {
+ "int",
+ "long",
+ "decimal",
+ "date",
+ "time",
+ "timestamp",
+ "timestamptz",
+ "timestamp_ns",
+ "timestamptz_ns",
+ "string",
+ "uuid",
+ "fixed",
+ "binary",
+}
+TRUNCATE_TYPES = {"int", "long", "decimal", "string", "binary"}
+DATE_TRANSFORM_TYPES = {
+ "date",
+ "timestamp",
+ "timestamptz",
+ "timestamp_ns",
+ "timestamptz_ns",
+}
+HOUR_TYPES = {"timestamp", "timestamptz", "timestamp_ns", "timestamptz_ns"}
+VOID_TYPES = IDENTITY_TYPES | {"geometry", "geography"}
+
+
+def _type_family(type_name):
+ if not isinstance(type_name, str):
+ return None
+ if type_name.startswith("decimal("):
+ return "decimal"
+ if type_name.startswith("fixed["):
+ return "fixed"
+ if type_name.startswith("geometry("):
+ return "geometry"
+ if type_name.startswith("geography("):
+ return "geography"
+ return type_name
+
+
+def _transform_family(transform):
+ if not isinstance(transform, str):
+ return None
+ if transform.startswith("bucket["):
+ return "bucket"
+ if transform.startswith("truncate["):
+ return "truncate"
+ return transform
+
+
+def _json_scalar_equal(left, right):
+ return type(left) is type(right) and left == right
+
+
+def _partition_transform_errors(path, doc, check_bucket=True):
+ errors = []
+ cases = doc.get("cases") if isinstance(doc, dict) else None
+ if not isinstance(cases, list):
+ return errors
+
+ allowed_sources = {
+ "identity": IDENTITY_TYPES,
+ "bucket": BUCKET_TYPES,
+ "truncate": TRUNCATE_TYPES,
+ "year": DATE_TRANSFORM_TYPES,
+ "month": DATE_TRANSFORM_TYPES,
+ "day": DATE_TRANSFORM_TYPES,
+ "hour": HOUR_TYPES,
+ "void": VOID_TYPES,
+ }
+
+ for index, case in enumerate(cases):
+ if not isinstance(case, dict):
+ continue
+ input_value = case.get("input")
+ decoded = case.get("decoded")
+ if not isinstance(input_value, dict) or not isinstance(decoded, dict):
+ continue
+
+ transform = input_value.get("transform")
+ source_type = input_value.get("source_type")
+ values = input_value.get("values")
+ result_types = decoded.get("result_types")
+ expected = decoded.get("values")
+ transform_family = _transform_family(transform)
+ source_family = _type_family(source_type)
+ label = f"{path}: cases/{index}"
+
+ permitted = allowed_sources.get(transform_family)
+ if permitted is None:
+ errors.append(f"{label}: unknown transform '{transform}'")
+ elif source_family not in permitted:
+ errors.append(
+ f"{label}: transform '{transform}' does not support source type "
+ f"'{source_type}'"
+ )
+
+ if isinstance(values, list) and isinstance(expected, list):
+ if len(values) != len(expected):
+ errors.append(
+ f"{label}: input.values and decoded.values must have equal lengths"
+ )
+ if None not in values:
+ errors.append(f"{label}: input.values must include null")
+ for value_index, (value, output) in enumerate(zip(values, expected)):
+ if value is None and output is not None:
+ errors.append(
+ f"{label}: decoded.values/{value_index} must be null for a "
+ "null input"
+ )
+ if transform_family == "void" and output is not None:
+ errors.append(
+ f"{label}: decoded.values/{value_index} must be null for void"
+ )
+ if transform_family == "identity" and not _json_scalar_equal(
+ value, output
+ ):
+ errors.append(
+ f"{label}: decoded.values/{value_index} must equal its "
+ "identity input"
+ )
+
+ required_result_types = None
+ if transform_family in {"bucket", "year", "month", "hour"}:
+ required_result_types = {"int"}
+ elif transform_family == "day":
+ required_result_types = {"date"}
+ elif transform_family in {"identity", "truncate"} and isinstance(
+ source_type, str
+ ):
+ required_result_types = {source_type}
+ elif transform_family == "void" and isinstance(source_type, str):
+ required_result_types = {source_type, "int"}
+
+ if (
+ required_result_types is not None
+ and isinstance(result_types, list)
+ and all(isinstance(result_type, str) for result_type in result_types)
+ ):
+ if set(result_types) != required_result_types:
+ errors.append(
+ f"{label}: decoded.result_types must be exactly "
+ f"{sorted(required_result_types)}"
+ )
+
+ bucket_path = path.replace("\\", "/").endswith(
+ "/partition-transforms/bucket/cases.json"
+ )
+ if check_bucket and not errors and bucket_path:
+ errors.extend(_bucket_errors(path, doc))
+
+ return errors
+
+
+EPOCH = datetime(1970, 1, 1, tzinfo=timezone.utc)
+TIMESTAMP = re.compile(
+ r"^(\d{4}-\d\d-\d\dT\d\d:\d\d:\d\d)(?:\.(\d{1,9}))?(Z|[+-]\d\d:\d\d)?$"
+)
+MASK = 0xFFFFFFFF
+
+
+def murmur3_x86_32(data):
+ """MurmurHash3 x86 32-bit with seed 0, returned as an unsigned integer."""
+ h = 0
+ for offset in range(0, len(data) // 4 * 4, 4):
+ k = int.from_bytes(data[offset : offset + 4], "little")
+ k = (k * 0xCC9E2D51) & MASK
+ k = ((k << 15) | (k >> 17)) & MASK
+ k = (k * 0x1B873593) & MASK
+ h ^= k
+ h = ((h << 13) | (h >> 19)) & MASK
+ h = (h * 5 + 0xE6546B64) & MASK
+
+ tail = data[len(data) // 4 * 4 :]
+ if tail:
+ k = int.from_bytes(tail, "little")
+ k = (k * 0xCC9E2D51) & MASK
+ k = ((k << 15) | (k >> 17)) & MASK
+ k = (k * 0x1B873593) & MASK
+ h ^= k
+
+ h ^= len(data)
+ h ^= h >> 16
+ h = (h * 0x85EBCA6B) & MASK
+ h ^= h >> 13
+ h = (h * 0xC2B2AE35) & MASK
+ return (h ^ (h >> 16)) & MASK
+
+
+def epoch_units(value, nanos=False):
+ match = TIMESTAMP.fullmatch(value)
+ if not match:
+ raise ValueError(f"invalid timestamp: {value}")
+ base, fraction, zone = match.groups()
+ instant = datetime.fromisoformat(base + ("+00:00" if zone == "Z" else zone or ""))
+ instant = instant.astimezone(timezone.utc) if zone else instant.replace(tzinfo=timezone.utc)
+ delta = instant - EPOCH
+ seconds = delta.days * 86400 + delta.seconds
+ digits = (fraction or "").ljust(9, "0")
+ nanoseconds = seconds * 1_000_000_000 + int(digits or "0")
+ if nanos and nanoseconds < 0 and nanoseconds % 1000:
+ raise ValueError("negative nanoseconds between microseconds need a spec rounding rule")
+ return nanoseconds // 1000
+
+
+def decimal_bytes(value):
+ decimal = Decimal(value)
+ digits = decimal.as_tuple()
+ unscaled = int("".join(map(str, digits.digits)))
+ if digits.sign:
+ unscaled = -unscaled
+ length = 1
+ while not -(1 << (length * 8 - 1)) <= unscaled < (1 << (length * 8 - 1)):
+ length += 1
+ return unscaled.to_bytes(length, "big", signed=True)
+
+
+def encode_for_hash(source_type, value):
+ if source_type in ("int", "long"):
+ return struct.pack("<q", int(value))
+ if source_type.startswith("decimal("):
+ return decimal_bytes(value)
+ if source_type == "date":
+ days = (date.fromisoformat(value) - date(1970, 1, 1)).days
+ return struct.pack("<q", days)
+ if source_type == "time":
+ clock = time.fromisoformat(value)
+ micros = ((clock.hour * 60 + clock.minute) * 60 + clock.second) * 1_000_000
+ return struct.pack("<q", micros + clock.microsecond)
+ if source_type in ("timestamp", "timestamptz", "timestamp_ns", "timestamptz_ns"):
+ return struct.pack("<q", epoch_units(value, source_type.endswith("_ns")))
+ if source_type == "string":
+ return value.encode("utf-8")
+ if source_type == "uuid":
+ return uuid.UUID(value).bytes
+ if source_type == "binary" or source_type.startswith("fixed["):
+ return bytes.fromhex(value)
+ raise ValueError(f"unsupported bucket source type: {source_type}")
+
+
+def _bucket_errors(path, doc):
+ vectors = (
+ (struct.pack("<q", 34), 2017239379),
+ (struct.pack("<q", 17486), -653330422 & MASK),
+ (b"iceberg", 1210000089),
+ )
+ for data, expected_hash in vectors:
+ if murmur3_x86_32(data) != expected_hash:
+ raise AssertionError("Murmur3 does not reproduce an Appendix B vector")
+
+ errors = []
+ for case_index, case in enumerate(doc["cases"]):
+ width = re.fullmatch(r"bucket\[(\d+)\]", case["input"]["transform"])
+ if not width or int(width[1]) <= 0:
+ errors.append(f"{path}: cases/{case_index}: invalid bucket transform")
+ continue
+ count = int(width[1])
+ source_type = case["input"]["source_type"]
+ values = case["input"]["values"]
+ expected = case["decoded"]["values"]
+ for index, (value, wanted) in enumerate(zip(values, expected)):
+ try:
+ actual = None
+ if value is not None:
+ hashed = murmur3_x86_32(encode_for_hash(source_type, value))
+ actual = (hashed & 0x7FFFFFFF) % count
+ except (ValueError, OverflowError, struct.error) as exc:
Review Comment:
A value that doesn't match `source_type` crashes the validator here instead of producing a `FAILED:` report. With `"source_type": "string", "values": [null, 5]` under `bucket/`, the run exits 1 with `AttributeError: 'int' object has no attribute 'encode'` and no `cases/N` location; decimal, date, time, timestamp, uuid and binary behave the same. The [selftest contract](https://github.com/apache/iceberg-verification/blob/a45307a7e7fa019fc339d03ba317387038030b12/Makefile#L52-L55) asks for a FAILED report, not a crash.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Add partition transform fixtures [iceberg-verification]
Posted by "romandenysov-akv (via GitHub) via issues" <is...@iceberg.apache.org>.
romandenysov-akv commented on PR #17:
URL: https://github.com/apache/iceberg-verification/pull/17#issuecomment-6825285586
> `dev/validate-fixtures.py` now recomputes every bucket value, and the README example has `null` first, thanks. Could you open an issue for recomputing truncate and the temporal transforms? The bucket-coverage follow-up could also take pre-epoch year/month/hour points at 1969-01-01, 1968-12-31 and 1969-12-31T23:00:00, which catch a truncate-then-subtract-one implementation. The validator notes inline are yours to handle here or in a follow-up.
@laskoviymishka
Thanks for the suggestion. I opened #23 to track the truncate and temporal validation and the additional pre-epoch cases.
The other comments resolved, it's lgtm to finish PR from my perspective.
Pls let me know if I missed something.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Add partition transform fixtures [iceberg-verification]
Posted by "laskoviymishka (via GitHub) via issues" <is...@iceberg.apache.org>.
laskoviymishka merged PR #17:
URL: https://github.com/apache/iceberg-verification/pull/17
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org