You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "loicalleyne (via GitHub)" <gi...@apache.org> on 2025/05/08 19:47:14 UTC
[PR] fix(table) getPartitionRecord to use field ID rather than field name [iceberg-go]
loicalleyne opened a new pull request, #427:
URL: https://github.com/apache/iceberg-go/pull/427
supersedes [#412](https://github.com/apache/iceberg-go/pull/412)
Changes:
- `getPartitionRecord` uses field ID rather than field Name to look up partition (addresses issue [404](https://github.com/apache/iceberg-go/issues/404) where java implementation is sanitizing name and causing mismatch)
- Breaking change - NewDataFileBuilder signature change
from:
```go
func NewDataFileBuilder(
spec PartitionSpec,
content ManifestEntryContent,
path string,
format FileFormat,
partitionData map[string]any,
recordCount int64,
fileSize int64,
) (*DataFileBuilder, error)
```
to:
```go
func NewDataFileBuilder(
spec PartitionSpec,
content ManifestEntryContent,
path string,
format FileFormat,
partitionData map[string]any,
fieldIDToPartitionData map[int]any,
recordCount int64,
fileSize int64,
) (*DataFileBuilder, error)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] fix(table) getPartitionRecord to use field ID rather than field name [iceberg-go]
Posted by "loicalleyne (via GitHub)" <gi...@apache.org>.
loicalleyne commented on code in PR #427:
URL: https://github.com/apache/iceberg-go/pull/427#discussion_r2087170899
##########
manifest.go:
##########
@@ -1480,6 +1479,33 @@ func avroPartitionData(input map[string]any, nameToID map[string]int, logicalTyp
return out
}
+func avroFieldPartitionData(input map[int]any, logicalTypes map[int]avro.LogicalType) map[int]any {
+ out := make(map[int]any)
+ for k, v := range input {
+ if logical, ok := logicalTypes[k]; ok {
+ switch logical {
+ case avro.Date:
+ out[k] = Date(v.(time.Time).Truncate(24*time.Hour).Unix() / int64((time.Hour * 24).Seconds()))
+ case avro.TimeMillis:
+ out[k] = Time(v.(time.Duration).Milliseconds())
+ case avro.TimeMicros:
+ out[k] = Time(v.(time.Duration).Microseconds())
+ case avro.TimestampMillis:
+ out[k] = Timestamp(v.(time.Time).UTC().UnixMilli())
+ case avro.TimestampMicros:
+ out[k] = Timestamp(v.(time.Time).UTC().UnixMicro())
+ default:
+ out[k] = v
+ }
+
+ continue
+ }
+ out[k] = v
+ }
+
+ return out
+}
Review Comment:
@zeroshade
`dataFile.Partition()` still returns `dataFile.PartitionData`, even though it's not used in `getPartitionRecord` anymore, we might still want to swap out the decoded types for iceberg types, what do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] fix(table) getPartitionRecord to use field ID rather than field name [iceberg-go]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #427:
URL: https://github.com/apache/iceberg-go/pull/427#discussion_r2087331400
##########
manifest.go:
##########
@@ -1480,6 +1479,33 @@ func avroPartitionData(input map[string]any, nameToID map[string]int, logicalTyp
return out
}
+func avroFieldPartitionData(input map[int]any, logicalTypes map[int]avro.LogicalType) map[int]any {
+ out := make(map[int]any)
+ for k, v := range input {
+ if logical, ok := logicalTypes[k]; ok {
+ switch logical {
+ case avro.Date:
+ out[k] = Date(v.(time.Time).Truncate(24*time.Hour).Unix() / int64((time.Hour * 24).Seconds()))
+ case avro.TimeMillis:
+ out[k] = Time(v.(time.Duration).Milliseconds())
+ case avro.TimeMicros:
+ out[k] = Time(v.(time.Duration).Microseconds())
+ case avro.TimestampMillis:
+ out[k] = Timestamp(v.(time.Time).UTC().UnixMilli())
+ case avro.TimestampMicros:
+ out[k] = Timestamp(v.(time.Time).UTC().UnixMicro())
+ default:
+ out[k] = v
+ }
+
+ continue
+ }
+ out[k] = v
+ }
+
+ return out
+}
Review Comment:
Yup please do. let's move all the functionality to use mapping of field-id -> data and remove the old version. We're good to merge this afterwards
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] fix(table) getPartitionRecord to use field ID rather than field name [iceberg-go]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on PR #427:
URL: https://github.com/apache/iceberg-go/pull/427#issuecomment-2867354420
> Perhaps we can add a test for https://github.com/apache/iceberg-go/issues/404, similar to this comment https://github.com/apache/iceberg-go/issues/404#issuecomment-2851945226
Agreed! I'd be fine with the test case being added in a subsequent PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] fix(table) getPartitionRecord to use field ID rather than field name [iceberg-go]
Posted by "loicalleyne (via GitHub)" <gi...@apache.org>.
loicalleyne commented on code in PR #427:
URL: https://github.com/apache/iceberg-go/pull/427#discussion_r2087312375
##########
manifest.go:
##########
@@ -1480,6 +1479,33 @@ func avroPartitionData(input map[string]any, nameToID map[string]int, logicalTyp
return out
}
+func avroFieldPartitionData(input map[int]any, logicalTypes map[int]avro.LogicalType) map[int]any {
+ out := make(map[int]any)
+ for k, v := range input {
+ if logical, ok := logicalTypes[k]; ok {
+ switch logical {
+ case avro.Date:
+ out[k] = Date(v.(time.Time).Truncate(24*time.Hour).Unix() / int64((time.Hour * 24).Seconds()))
+ case avro.TimeMillis:
+ out[k] = Time(v.(time.Duration).Milliseconds())
+ case avro.TimeMicros:
+ out[k] = Time(v.(time.Duration).Microseconds())
+ case avro.TimestampMillis:
+ out[k] = Timestamp(v.(time.Time).UTC().UnixMilli())
+ case avro.TimestampMicros:
+ out[k] = Timestamp(v.(time.Time).UTC().UnixMicro())
+ default:
+ out[k] = v
+ }
+
+ continue
+ }
+ out[k] = v
+ }
+
+ return out
+}
Review Comment:
If we're agreed on stripping the field name implementation out completely I'll remove `avroFieldPartitionData` and `dataFile.PartitionFieldData`, and move the functionality of `dataFile.PartitionFieldData` to `dataFile.Partition` - clearer and more concise.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] fix(table) getPartitionRecord to use field ID rather than field name [iceberg-go]
Posted by "loicalleyne (via GitHub)" <gi...@apache.org>.
loicalleyne commented on code in PR #427:
URL: https://github.com/apache/iceberg-go/pull/427#discussion_r2085190744
##########
manifest.go:
##########
@@ -735,6 +739,7 @@ func (c *ManifestReader) ReadEntry() (ManifestEntry, error) {
tmp.inherit(c.file)
if fieldToIDMap, ok := tmp.DataFile().(hasFieldToIDMap); ok {
fieldToIDMap.setFieldNameToIDMap(c.fieldNameToID)
+ fieldToIDMap.setFieldIDToNameMap(c.fieldIDToName)
Review Comment:
done
##########
manifest_test.go:
##########
@@ -1074,7 +1074,7 @@ func (m *ManifestTestSuite) TestManifestEntryBuilder() {
EntryContentData,
"sample.parquet",
ParquetFile,
- map[string]any{"int": int(1), "datetime": time.Unix(1925, 0).UnixMicro()},
+ nil,
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] fix(table) getPartitionRecord to use field ID rather than field name [iceberg-go]
Posted by "GabrielM98 (via GitHub)" <gi...@apache.org>.
GabrielM98 commented on PR #427:
URL: https://github.com/apache/iceberg-go/pull/427#issuecomment-2865793295
Just tested this and it fixes my issue (#404). Thanks @loicalleyne!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] fix(table) getPartitionRecord to use field ID rather than field name [iceberg-go]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #427:
URL: https://github.com/apache/iceberg-go/pull/427#discussion_r2087145318
##########
manifest.go:
##########
@@ -1480,6 +1479,33 @@ func avroPartitionData(input map[string]any, nameToID map[string]int, logicalTyp
return out
}
+func avroFieldPartitionData(input map[int]any, logicalTypes map[int]avro.LogicalType) map[int]any {
+ out := make(map[int]any)
+ for k, v := range input {
+ if logical, ok := logicalTypes[k]; ok {
+ switch logical {
+ case avro.Date:
+ out[k] = Date(v.(time.Time).Truncate(24*time.Hour).Unix() / int64((time.Hour * 24).Seconds()))
+ case avro.TimeMillis:
+ out[k] = Time(v.(time.Duration).Milliseconds())
+ case avro.TimeMicros:
+ out[k] = Time(v.(time.Duration).Microseconds())
+ case avro.TimestampMillis:
+ out[k] = Timestamp(v.(time.Time).UTC().UnixMilli())
+ case avro.TimestampMicros:
+ out[k] = Timestamp(v.(time.Time).UTC().UnixMicro())
+ default:
+ out[k] = v
+ }
+
+ continue
+ }
+ out[k] = v
+ }
+
+ return out
+}
Review Comment:
do we need the original `avroPartitionData` function anymore now that we have this new one?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] fix(table) getPartitionRecord to use field ID rather than field name [iceberg-go]
Posted by "loicalleyne (via GitHub)" <gi...@apache.org>.
loicalleyne commented on code in PR #427:
URL: https://github.com/apache/iceberg-go/pull/427#discussion_r2085193689
##########
manifest.go:
##########
@@ -415,31 +415,33 @@ func getFieldIDMap(sc avro.Schema) (map[string]int, map[int]avro.LogicalType) {
}
result := make(map[string]int)
+ fieldIDToName := make(map[int]string)
Review Comment:
removed from `getFieldIDMap()`, left the field in `dataFile` but populate it in `setFieldNameToIDMap`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] fix(table) getPartitionRecord to use field ID rather than field name [iceberg-go]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #427:
URL: https://github.com/apache/iceberg-go/pull/427#discussion_r2082119647
##########
manifest.go:
##########
@@ -735,6 +739,7 @@ func (c *ManifestReader) ReadEntry() (ManifestEntry, error) {
tmp.inherit(c.file)
if fieldToIDMap, ok := tmp.DataFile().(hasFieldToIDMap); ok {
fieldToIDMap.setFieldNameToIDMap(c.fieldNameToID)
+ fieldToIDMap.setFieldIDToNameMap(c.fieldIDToName)
Review Comment:
we could probably just have `setFieldNameToIDMap` automatically generate the inverse map (or vice versa) rather than having to generate both maps in `getFieldIDMap`
##########
manifest.go:
##########
@@ -1480,6 +1485,33 @@ func avroPartitionData(input map[string]any, nameToID map[string]int, logicalTyp
return out
}
+func avroFieldPartitionData(input map[int]any, idToName map[int]string, logicalTypes map[int]avro.LogicalType) map[int]any {
Review Comment:
the `idToName` map isn't being used at all, we should probably remove that as an argument
##########
manifest.go:
##########
@@ -1527,11 +1561,22 @@ func (d *dataFile) initializeMapData() {
d.distinctCntMap = avroColMapToMap(d.DistinctCounts)
d.lowerBoundMap = avroColMapToMap(d.LowerBounds)
d.upperBoundMap = avroColMapToMap(d.UpperBounds)
+ // dataFile read from avro
+ if len(d.fieldIDToPartitionData) < len(d.PartitionData) {
+ d.fieldIDToPartitionData = make(map[int]any, len(d.PartitionData))
+ for k, v := range d.PartitionData {
+ if id, ok := d.fieldNameToID[k]; ok {
+ d.fieldIDToPartitionData[id] = v
+ }
+ }
+ }
d.PartitionData = avroPartitionData(d.PartitionData, d.fieldNameToID, d.fieldIDToLogicalType)
+ d.fieldIDToPartitionData = avroFieldPartitionData(d.fieldIDToPartitionData, d.fieldIDToName, d.fieldIDToLogicalType)
Review Comment:
could we pass the NameToID map into `avroPartitionData` and just do the mapping there?
##########
manifest.go:
##########
@@ -415,31 +415,33 @@ func getFieldIDMap(sc avro.Schema) (map[string]int, map[int]avro.LogicalType) {
}
result := make(map[string]int)
+ fieldIDToName := make(map[int]string)
Review Comment:
do we really need both of these maps? especially given that one is just the inverse of the other.
##########
manifest_test.go:
##########
@@ -1074,7 +1074,7 @@ func (m *ManifestTestSuite) TestManifestEntryBuilder() {
EntryContentData,
"sample.parquet",
ParquetFile,
- map[string]any{"int": int(1), "datetime": time.Unix(1925, 0).UnixMicro()},
+ nil,
Review Comment:
instead of changing this to nil, we should be using the correct mapping here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] fix(table) getPartitionRecord to use field ID rather than field name [iceberg-go]
Posted by "loicalleyne (via GitHub)" <gi...@apache.org>.
loicalleyne commented on code in PR #427:
URL: https://github.com/apache/iceberg-go/pull/427#discussion_r2085472448
##########
manifest.go:
##########
@@ -415,31 +415,33 @@ func getFieldIDMap(sc avro.Schema) (map[string]int, map[int]avro.LogicalType) {
}
result := make(map[string]int)
+ fieldIDToName := make(map[int]string)
Review Comment:
ah well the linter doesn't like it , I'll remove it completely
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] fix(table) getPartitionRecord to use field ID rather than field name [iceberg-go]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on code in PR #427:
URL: https://github.com/apache/iceberg-go/pull/427#discussion_r2087170419
##########
manifest.go:
##########
@@ -1480,6 +1479,33 @@ func avroPartitionData(input map[string]any, nameToID map[string]int, logicalTyp
return out
}
+func avroFieldPartitionData(input map[int]any, logicalTypes map[int]avro.LogicalType) map[int]any {
+ out := make(map[int]any)
+ for k, v := range input {
+ if logical, ok := logicalTypes[k]; ok {
+ switch logical {
+ case avro.Date:
+ out[k] = Date(v.(time.Time).Truncate(24*time.Hour).Unix() / int64((time.Hour * 24).Seconds()))
+ case avro.TimeMillis:
+ out[k] = Time(v.(time.Duration).Milliseconds())
+ case avro.TimeMicros:
+ out[k] = Time(v.(time.Duration).Microseconds())
+ case avro.TimestampMillis:
+ out[k] = Timestamp(v.(time.Time).UTC().UnixMilli())
+ case avro.TimestampMicros:
+ out[k] = Timestamp(v.(time.Time).UTC().UnixMicro())
+ default:
+ out[k] = v
+ }
+
+ continue
+ }
+ out[k] = v
+ }
+
+ return out
+}
Review Comment:
we might want to clean up any old implementations thats using field name as its not the recommended way to access partition data. For example, `DataFile.Partition()`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org