You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "arnaudbriche (via GitHub)" <gi...@apache.org> on 2025/02/17 15:55:43 UTC
[PR] Fix #305: ManifestEntry partition field schema should be dynamically … [iceberg-go]
arnaudbriche opened a new pull request, #307:
URL: https://github.com/apache/iceberg-go/pull/307
Avro schemas for Iceberg metadata files are now built dynamically with code.
I know this changes a lot of code, but I did not find other less intrusive ways to fix the issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Fix #305: ManifestEntry partition field schema should be dynamically … [iceberg-go]
Posted by "arnaudbriche (via GitHub)" <gi...@apache.org>.
arnaudbriche commented on code in PR #307:
URL: https://github.com/apache/iceberg-go/pull/307#discussion_r1960562598
##########
utils.go:
##########
@@ -238,3 +239,68 @@ func avroEncode[T any](key string, version int, vals []T, out io.Writer) error {
return enc.Close()
}
+
+func structTypeToAvroPartitionSchema(st *StructType) (avro.Schema, error) {
+ var aFields []*avro.Field
+
+ for _, field := range st.Fields() {
+ aField, err := nestedFieldToAvroField(field)
+
+ if err != nil {
+ return nil, err
+ }
+
+ aFields = append(aFields, aField)
+ }
+
+ return avro.NewRecordSchema("r102", "", aFields)
Review Comment:
Here is how Pyiceberg derive the record name: https://github.com/apache/iceberg-python/blob/efc8b5ac0f16717f776e034ecf9a9e9bdabd8424/pyiceberg/utils/schema_conversion.py#L543
##########
utils.go:
##########
@@ -238,3 +239,68 @@ func avroEncode[T any](key string, version int, vals []T, out io.Writer) error {
return enc.Close()
}
+
+func structTypeToAvroPartitionSchema(st *StructType) (avro.Schema, error) {
+ var aFields []*avro.Field
+
+ for _, field := range st.Fields() {
+ aField, err := nestedFieldToAvroField(field)
+
+ if err != nil {
+ return nil, err
+ }
+
+ aFields = append(aFields, aField)
+ }
+
+ return avro.NewRecordSchema("r102", "", aFields)
Review Comment:
Here is how Pyiceberg derives the record name: https://github.com/apache/iceberg-python/blob/efc8b5ac0f16717f776e034ecf9a9e9bdabd8424/pyiceberg/utils/schema_conversion.py#L543
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Fix #305: ManifestEntry partition field schema should be dynamically … [iceberg-go]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on PR #307:
URL: https://github.com/apache/iceberg-go/pull/307#issuecomment-2675784737
Ah i had to accept the invite
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Fix #305: ManifestEntry partition field schema should be dynamically … [iceberg-go]
Posted by "arnaudbriche (via GitHub)" <gi...@apache.org>.
arnaudbriche commented on code in PR #307:
URL: https://github.com/apache/iceberg-go/pull/307#discussion_r1960767524
##########
utils.go:
##########
@@ -238,3 +239,68 @@ func avroEncode[T any](key string, version int, vals []T, out io.Writer) error {
return enc.Close()
}
+
+func structTypeToAvroPartitionSchema(st *StructType) (avro.Schema, error) {
+ var aFields []*avro.Field
+
+ for _, field := range st.Fields() {
+ aField, err := nestedFieldToAvroField(field)
+
+ if err != nil {
+ return nil, err
+ }
+
+ aFields = append(aFields, aField)
+ }
+
+ return avro.NewRecordSchema("r102", "", aFields)
+}
+
+func nestedFieldToAvroField(f NestedField) (*avro.Field, error) {
+ sch, err := nestedFieldToAvroSchema(f)
+
+ if err != nil {
+ return nil, err
+ }
+
+ return avro.NewField(
+ f.Name,
+ sch,
+ avro.WithDoc(f.Doc),
+ avro.WithProps(map[string]any{"field-id": f.ID}),
+ )
+}
+
+func nestedFieldToAvroSchema(f NestedField) (avro.Schema, error) {
+ var sch avro.Schema
+
+ switch f.Type.(type) {
+ case *StringType:
+ sch = avro.NewPrimitiveSchema(avro.String, nil)
+ case *Int32Type:
+ sch = avro.NewPrimitiveSchema(avro.Int, nil)
+ case *Int64Type:
+ sch = avro.NewPrimitiveSchema(avro.Long, nil)
+ case *BinaryType:
+ sch = avro.NewPrimitiveSchema(avro.Bytes, nil)
+ case *BooleanType:
+ sch = avro.NewPrimitiveSchema(avro.Boolean, nil)
+ case *Float32Type:
+ sch = avro.NewPrimitiveSchema(avro.Float, nil)
+ case *Float64Type:
+ sch = avro.NewPrimitiveSchema(avro.Double, nil)
+ case *DateType:
+ sch = avro.NewPrimitiveSchema(avro.Int, avro.NewPrimitiveLogicalSchema(avro.Date))
+ default:
Review Comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Fix #305: ManifestEntry partition field schema should be dynamically … [iceberg-go]
Posted by "arnaudbriche (via GitHub)" <gi...@apache.org>.
arnaudbriche commented on code in PR #307:
URL: https://github.com/apache/iceberg-go/pull/307#discussion_r1960769877
##########
manifest.go:
##########
@@ -292,10 +307,15 @@ func (b *ManifestV2Builder) KeyMetadata(km []byte) *ManifestV2Builder {
// a pointer to the constructed manifest file. Further calls to the modifier
// methods after calling build would modify the constructed ManifestFile.
func (b *ManifestV2Builder) Build() ManifestFile {
+ if b.m.partitionType == nil {
+ b.m.partitionType = UnpartitionedSpec.PartitionType(nil)
+ }
+
return b.m
}
type manifestFileV2 struct {
+ partitionType *StructType
Review Comment:
I have done `Avro` <--> `Iceberg` schema conversions.
Can you please help me with the populate part ? Maybe point me to the right part of the code ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Fix #305: ManifestEntry partition field schema should be dynamically … [iceberg-go]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #307:
URL: https://github.com/apache/iceberg-go/pull/307#discussion_r1960182073
##########
utils.go:
##########
@@ -238,3 +239,68 @@ func avroEncode[T any](key string, version int, vals []T, out io.Writer) error {
return enc.Close()
}
+
+func structTypeToAvroPartitionSchema(st *StructType) (avro.Schema, error) {
+ var aFields []*avro.Field
+
+ for _, field := range st.Fields() {
+ aField, err := nestedFieldToAvroField(field)
+
+ if err != nil {
+ return nil, err
+ }
+
+ aFields = append(aFields, aField)
+ }
+
+ return avro.NewRecordSchema("r102", "", aFields)
Review Comment:
@kevinjqliu @Fokko @nastra any of you happen to know there answer on this one?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Fix #305: ManifestEntry partition field schema should be dynamically … [iceberg-go]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on PR #307:
URL: https://github.com/apache/iceberg-go/pull/307#issuecomment-2675865197
The only way to get the file length from the file itself is via Seeking to the end unfortunately.
ReadAt could be sufficient, but would require having to externally provide the file's size somehow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Fix #305: ManifestEntry partition field schema should be dynamically … [iceberg-go]
Posted by "arnaudbriche (via GitHub)" <gi...@apache.org>.
arnaudbriche commented on PR #307:
URL: https://github.com/apache/iceberg-go/pull/307#issuecomment-2675751583
@zeroshade
I cannot find the "Allow edits from maintainers" button. Can you point-out where it is supposed to be ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Fix #305: ManifestEntry partition field schema should be dynamically … [iceberg-go]
Posted by "arnaudbriche (via GitHub)" <gi...@apache.org>.
arnaudbriche commented on code in PR #307:
URL: https://github.com/apache/iceberg-go/pull/307#discussion_r1959518433
##########
manifest.go:
##########
@@ -292,10 +307,15 @@ func (b *ManifestV2Builder) KeyMetadata(km []byte) *ManifestV2Builder {
// a pointer to the constructed manifest file. Further calls to the modifier
// methods after calling build would modify the constructed ManifestFile.
func (b *ManifestV2Builder) Build() ManifestFile {
+ if b.m.partitionType == nil {
+ b.m.partitionType = UnpartitionedSpec.PartitionType(nil)
+ }
+
return b.m
}
type manifestFileV2 struct {
+ partitionType *StructType
Review Comment:
Not sure if it belongs to `FieldSummary`.
We also need to be able to do `Avro` -> `Iceberg` schema conversion to populate the field on read.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Fix #305: ManifestEntry partition field schema should be dynamically … [iceberg-go]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on PR #307:
URL: https://github.com/apache/iceberg-go/pull/307#issuecomment-2675786497
@arnaudbriche can you take a look at my updated version here and make sure that it works with what you were trying for clickhouse? Feel free to comment on the actual API also if you like
@Fokko @kevinjqliu I'd also love any feedback you two might have as far as the manifest writing APIs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Fix #305: ManifestEntry partition field schema should be dynamically … [iceberg-go]
Posted by "arnaudbriche (via GitHub)" <gi...@apache.org>.
arnaudbriche commented on PR #307:
URL: https://github.com/apache/iceberg-go/pull/307#issuecomment-2675851774
Ok. I though ReadAt would be sufficient. Thx for answering!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Fix #305: ManifestEntry partition field schema should be dynamically … [iceberg-go]
Posted by "arnaudbriche (via GitHub)" <gi...@apache.org>.
arnaudbriche commented on PR #307:
URL: https://github.com/apache/iceberg-go/pull/307#issuecomment-2675759950
Seems like the feature does not exists when the fork repository is owner by an Organization.
I just gave you write access to the fork repository.
Is that ok ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Fix #305: ManifestEntry partition field schema should be dynamically … [iceberg-go]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #307:
URL: https://github.com/apache/iceberg-go/pull/307#discussion_r1960573870
##########
utils.go:
##########
@@ -238,3 +239,68 @@ func avroEncode[T any](key string, version int, vals []T, out io.Writer) error {
return enc.Close()
}
+
+func structTypeToAvroPartitionSchema(st *StructType) (avro.Schema, error) {
+ var aFields []*avro.Field
+
+ for _, field := range st.Fields() {
+ aField, err := nestedFieldToAvroField(field)
+
+ if err != nil {
+ return nil, err
+ }
+
+ aFields = append(aFields, aField)
+ }
+
+ return avro.NewRecordSchema("r102", "", aFields)
Review Comment:
seems reasonable we should do the same then :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Fix #305: ManifestEntry partition field schema should be dynamically … [iceberg-go]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #307:
URL: https://github.com/apache/iceberg-go/pull/307#discussion_r1960181193
##########
utils.go:
##########
@@ -238,3 +239,68 @@ func avroEncode[T any](key string, version int, vals []T, out io.Writer) error {
return enc.Close()
}
+
+func structTypeToAvroPartitionSchema(st *StructType) (avro.Schema, error) {
+ var aFields []*avro.Field
+
+ for _, field := range st.Fields() {
+ aField, err := nestedFieldToAvroField(field)
+
+ if err != nil {
+ return nil, err
+ }
+
+ aFields = append(aFields, aField)
+ }
+
+ return avro.NewRecordSchema("r102", "", aFields)
+}
+
+func nestedFieldToAvroField(f NestedField) (*avro.Field, error) {
+ sch, err := nestedFieldToAvroSchema(f)
+
+ if err != nil {
+ return nil, err
+ }
+
+ return avro.NewField(
+ f.Name,
+ sch,
+ avro.WithDoc(f.Doc),
+ avro.WithProps(map[string]any{"field-id": f.ID}),
+ )
+}
+
+func nestedFieldToAvroSchema(f NestedField) (avro.Schema, error) {
+ var sch avro.Schema
+
+ switch f.Type.(type) {
+ case *StringType:
+ sch = avro.NewPrimitiveSchema(avro.String, nil)
+ case *Int32Type:
+ sch = avro.NewPrimitiveSchema(avro.Int, nil)
+ case *Int64Type:
+ sch = avro.NewPrimitiveSchema(avro.Long, nil)
+ case *BinaryType:
+ sch = avro.NewPrimitiveSchema(avro.Bytes, nil)
+ case *BooleanType:
+ sch = avro.NewPrimitiveSchema(avro.Boolean, nil)
+ case *Float32Type:
+ sch = avro.NewPrimitiveSchema(avro.Float, nil)
+ case *Float64Type:
+ sch = avro.NewPrimitiveSchema(avro.Double, nil)
+ case *DateType:
+ sch = avro.NewPrimitiveSchema(avro.Int, avro.NewPrimitiveLogicalSchema(avro.Date))
+ default:
Review Comment:
As far as I'm aware, yes. You can't partition on a struct/list/map type to my knowledge.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Fix #305: ManifestEntry partition field schema should be dynamically … [iceberg-go]
Posted by "arnaudbriche (via GitHub)" <gi...@apache.org>.
arnaudbriche commented on code in PR #307:
URL: https://github.com/apache/iceberg-go/pull/307#discussion_r1959422260
##########
utils.go:
##########
@@ -238,3 +239,68 @@ func avroEncode[T any](key string, version int, vals []T, out io.Writer) error {
return enc.Close()
}
+
+func structTypeToAvroPartitionSchema(st *StructType) (avro.Schema, error) {
+ var aFields []*avro.Field
+
+ for _, field := range st.Fields() {
+ aField, err := nestedFieldToAvroField(field)
+
+ if err != nil {
+ return nil, err
+ }
+
+ aFields = append(aFields, aField)
+ }
+
+ return avro.NewRecordSchema("r102", "", aFields)
Review Comment:
Not sure. The spec seems not to enforce any specific name here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Fix #305: ManifestEntry partition field schema should be dynamically … [iceberg-go]
Posted by "arnaudbriche (via GitHub)" <gi...@apache.org>.
arnaudbriche commented on code in PR #307:
URL: https://github.com/apache/iceberg-go/pull/307#discussion_r1960768123
##########
utils.go:
##########
@@ -238,3 +239,68 @@ func avroEncode[T any](key string, version int, vals []T, out io.Writer) error {
return enc.Close()
}
+
+func structTypeToAvroPartitionSchema(st *StructType) (avro.Schema, error) {
+ var aFields []*avro.Field
+
+ for _, field := range st.Fields() {
+ aField, err := nestedFieldToAvroField(field)
+
+ if err != nil {
+ return nil, err
+ }
+
+ aFields = append(aFields, aField)
+ }
+
+ return avro.NewRecordSchema("r102", "", aFields)
+}
+
+func nestedFieldToAvroField(f NestedField) (*avro.Field, error) {
+ sch, err := nestedFieldToAvroSchema(f)
+
+ if err != nil {
+ return nil, err
+ }
+
+ return avro.NewField(
+ f.Name,
+ sch,
+ avro.WithDoc(f.Doc),
+ avro.WithProps(map[string]any{"field-id": f.ID}),
+ )
+}
+
+func nestedFieldToAvroSchema(f NestedField) (avro.Schema, error) {
+ var sch avro.Schema
+
+ switch f.Type.(type) {
+ case *StringType:
+ sch = avro.NewPrimitiveSchema(avro.String, nil)
+ case *Int32Type:
+ sch = avro.NewPrimitiveSchema(avro.Int, nil)
+ case *Int64Type:
+ sch = avro.NewPrimitiveSchema(avro.Long, nil)
+ case *BinaryType:
+ sch = avro.NewPrimitiveSchema(avro.Bytes, nil)
+ case *BooleanType:
+ sch = avro.NewPrimitiveSchema(avro.Boolean, nil)
+ case *Float32Type:
+ sch = avro.NewPrimitiveSchema(avro.Float, nil)
+ case *Float64Type:
+ sch = avro.NewPrimitiveSchema(avro.Double, nil)
+ case *DateType:
+ sch = avro.NewPrimitiveSchema(avro.Int, avro.NewPrimitiveLogicalSchema(avro.Date))
+ default:
Review Comment:
Let me know if you see missing ones.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Fix #305: ManifestEntry partition field schema should be dynamically … [iceberg-go]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on PR #307:
URL: https://github.com/apache/iceberg-go/pull/307#issuecomment-2675842086
The seek is necessary for efficient Parquet processing, and required by the interface APIs for reading Parquet
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Fix #305: ManifestEntry partition field schema should be dynamically … [iceberg-go]
Posted by "arnaudbriche (via GitHub)" <gi...@apache.org>.
arnaudbriche commented on code in PR #307:
URL: https://github.com/apache/iceberg-go/pull/307#discussion_r1959421586
##########
utils.go:
##########
@@ -238,3 +239,68 @@ func avroEncode[T any](key string, version int, vals []T, out io.Writer) error {
return enc.Close()
}
+
+func structTypeToAvroPartitionSchema(st *StructType) (avro.Schema, error) {
+ var aFields []*avro.Field
+
+ for _, field := range st.Fields() {
+ aField, err := nestedFieldToAvroField(field)
+
+ if err != nil {
+ return nil, err
+ }
+
+ aFields = append(aFields, aField)
+ }
+
+ return avro.NewRecordSchema("r102", "", aFields)
+}
+
+func nestedFieldToAvroField(f NestedField) (*avro.Field, error) {
+ sch, err := nestedFieldToAvroSchema(f)
+
+ if err != nil {
+ return nil, err
+ }
+
+ return avro.NewField(
+ f.Name,
+ sch,
+ avro.WithDoc(f.Doc),
+ avro.WithProps(map[string]any{"field-id": f.ID}),
+ )
+}
+
+func nestedFieldToAvroSchema(f NestedField) (avro.Schema, error) {
+ var sch avro.Schema
+
+ switch f.Type.(type) {
+ case *StringType:
+ sch = avro.NewPrimitiveSchema(avro.String, nil)
+ case *Int32Type:
+ sch = avro.NewPrimitiveSchema(avro.Int, nil)
+ case *Int64Type:
+ sch = avro.NewPrimitiveSchema(avro.Long, nil)
+ case *BinaryType:
+ sch = avro.NewPrimitiveSchema(avro.Bytes, nil)
+ case *BooleanType:
+ sch = avro.NewPrimitiveSchema(avro.Boolean, nil)
+ case *Float32Type:
+ sch = avro.NewPrimitiveSchema(avro.Float, nil)
+ case *Float64Type:
+ sch = avro.NewPrimitiveSchema(avro.Double, nil)
+ case *DateType:
+ sch = avro.NewPrimitiveSchema(avro.Int, avro.NewPrimitiveLogicalSchema(avro.Date))
+ default:
Review Comment:
Sure. Do we agree that we only need to handle primitive/scalar types here ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Fix #305: ManifestEntry partition field schema should be dynamically … [iceberg-go]
Posted by "arnaudbriche (via GitHub)" <gi...@apache.org>.
arnaudbriche commented on code in PR #307:
URL: https://github.com/apache/iceberg-go/pull/307#discussion_r1959424998
##########
internal/manifest-entry-v1.json:
##########
@@ -0,0 +1,182 @@
+{
Review Comment:
I just used them as an example to build the dynamic schema building code. I removed them now that they are no longer useful.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Fix #305: ManifestEntry partition field schema should be dynamically … [iceberg-go]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on PR #307:
URL: https://github.com/apache/iceberg-go/pull/307#issuecomment-2672549458
@arnaudbriche i'll take a look and see if I can condense and simplify this a bit for you, probably quicker than going back and forth like this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Fix #305: ManifestEntry partition field schema should be dynamically … [iceberg-go]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #307:
URL: https://github.com/apache/iceberg-go/pull/307#discussion_r1958843788
##########
utils.go:
##########
@@ -238,3 +239,68 @@ func avroEncode[T any](key string, version int, vals []T, out io.Writer) error {
return enc.Close()
}
+
+func structTypeToAvroPartitionSchema(st *StructType) (avro.Schema, error) {
+ var aFields []*avro.Field
+
+ for _, field := range st.Fields() {
+ aField, err := nestedFieldToAvroField(field)
+
+ if err != nil {
+ return nil, err
+ }
+
+ aFields = append(aFields, aField)
+ }
+
+ return avro.NewRecordSchema("r102", "", aFields)
+}
+
+func nestedFieldToAvroField(f NestedField) (*avro.Field, error) {
+ sch, err := nestedFieldToAvroSchema(f)
+
+ if err != nil {
+ return nil, err
+ }
+
+ return avro.NewField(
+ f.Name,
+ sch,
+ avro.WithDoc(f.Doc),
+ avro.WithProps(map[string]any{"field-id": f.ID}),
+ )
+}
+
+func nestedFieldToAvroSchema(f NestedField) (avro.Schema, error) {
+ var sch avro.Schema
+
+ switch f.Type.(type) {
+ case *StringType:
+ sch = avro.NewPrimitiveSchema(avro.String, nil)
+ case *Int32Type:
+ sch = avro.NewPrimitiveSchema(avro.Int, nil)
+ case *Int64Type:
+ sch = avro.NewPrimitiveSchema(avro.Long, nil)
+ case *BinaryType:
+ sch = avro.NewPrimitiveSchema(avro.Bytes, nil)
+ case *BooleanType:
+ sch = avro.NewPrimitiveSchema(avro.Boolean, nil)
+ case *Float32Type:
+ sch = avro.NewPrimitiveSchema(avro.Float, nil)
+ case *Float64Type:
+ sch = avro.NewPrimitiveSchema(avro.Double, nil)
+ case *DateType:
+ sch = avro.NewPrimitiveSchema(avro.Int, avro.NewPrimitiveLogicalSchema(avro.Date))
+ default:
Review Comment:
missing some types here
##########
internal/manifest_list_v1.go:
##########
@@ -0,0 +1,142 @@
+package internal
+
+import (
+ "github.com/hamba/avro/v2"
+)
+
+func MustNewManifestListV1Schema() avro.Schema {
+ return MustNewRecordSchema("manifest_file", "", []*avro.Field{
+ MustNewField(
+ "manifest_path",
+ avro.NewPrimitiveSchema(avro.String, nil),
+ avro.WithDoc("Location URI with FS scheme"),
+ avro.WithProps(map[string]any{"field-id": 500}),
+ ),
+ MustNewField(
+ "manifest_length",
+ avro.NewPrimitiveSchema(avro.Long, nil),
+ avro.WithDoc("Total file size in bytes"),
+ avro.WithProps(map[string]any{"field-id": 501}),
+ ),
+ MustNewField(
+ "partition_spec_id",
+ avro.NewPrimitiveSchema(avro.Int, nil),
+ avro.WithDoc("Spec ID used to write"),
+ avro.WithProps(map[string]any{"field-id": 502}),
+ ),
+ MustNewField(
+ "added_snapshot_id",
+ avro.NewPrimitiveSchema(avro.Long, nil),
+ avro.WithDoc("Snapshot ID that added the manifest"),
+ avro.WithProps(map[string]any{"field-id": 503}),
+ ),
+ MustNewField(
+ "added_data_files_count",
+ MustNewUnionSchema([]avro.Schema{
+ avro.NewNullSchema(),
+ avro.NewPrimitiveSchema(avro.Int, nil),
+ }),
+ avro.WithDoc("Added entry count"),
+ avro.WithProps(map[string]any{"field-id": 504}),
+ ),
+ MustNewField(
+ "existing_data_files_count",
+ MustNewUnionSchema([]avro.Schema{
+ avro.NewNullSchema(),
+ avro.NewPrimitiveSchema(avro.Int, nil),
+ }),
+ avro.WithDoc("Existing entry count"),
+ avro.WithProps(map[string]any{"field-id": 505}),
+ ),
+ MustNewField(
+ "deleted_data_files_count",
+ MustNewUnionSchema([]avro.Schema{
+ avro.NewNullSchema(),
+ avro.NewPrimitiveSchema(avro.Int, nil),
+ }),
+ avro.WithDoc("Deleted entry count"),
+ avro.WithProps(map[string]any{"field-id": 506}),
+ ),
+ MustNewField(
+ "partitions",
+ MustNewUnionSchema([]avro.Schema{
+ avro.NewNullSchema(),
+ avro.NewArraySchema(
+ MustNewPartitionSchema(),
+ avro.WithProps(map[string]any{"element-id": 508}),
+ ),
+ }),
+ avro.WithDoc("Summary for each partition"),
+ avro.WithProps(map[string]any{"field-id": 507}),
+ ),
+ MustNewField(
+ "added_rows_count",
+ MustNewUnionSchema([]avro.Schema{
+ avro.NewNullSchema(),
+ avro.NewPrimitiveSchema(avro.Long, nil),
+ }),
+ avro.WithDoc("Added rows count"),
+ avro.WithProps(map[string]any{"field-id": 512}),
+ ),
+ MustNewField(
+ "existing_rows_count",
+ MustNewUnionSchema([]avro.Schema{
+ avro.NewNullSchema(),
+ avro.NewPrimitiveSchema(avro.Long, nil),
+ }),
+ avro.WithDoc("Existing rows count"),
+ avro.WithProps(map[string]any{"field-id": 513}),
+ ),
+ MustNewField(
+ "deleted_rows_count",
+ MustNewUnionSchema([]avro.Schema{
+ avro.NewNullSchema(),
+ avro.NewPrimitiveSchema(avro.Long, nil),
+ }),
+ avro.WithDoc("Deleted rows count"),
+ avro.WithProps(map[string]any{"field-id": 514}),
+ ),
+ })
+}
+
+func MustNewPartitionSchema() avro.Schema {
Review Comment:
again, can this static schema be cached rather than built on demand
##########
manifest.go:
##########
@@ -292,10 +307,15 @@ func (b *ManifestV2Builder) KeyMetadata(km []byte) *ManifestV2Builder {
// a pointer to the constructed manifest file. Further calls to the modifier
// methods after calling build would modify the constructed ManifestFile.
func (b *ManifestV2Builder) Build() ManifestFile {
+ if b.m.partitionType == nil {
+ b.m.partitionType = UnpartitionedSpec.PartitionType(nil)
+ }
+
return b.m
}
type manifestFileV2 struct {
+ partitionType *StructType
Review Comment:
In either case, we should probably be populating this after reading in a manifest file too, right?
##########
manifest.go:
##########
@@ -292,10 +307,15 @@ func (b *ManifestV2Builder) KeyMetadata(km []byte) *ManifestV2Builder {
// a pointer to the constructed manifest file. Further calls to the modifier
// methods after calling build would modify the constructed ManifestFile.
func (b *ManifestV2Builder) Build() ManifestFile {
+ if b.m.partitionType == nil {
+ b.m.partitionType = UnpartitionedSpec.PartitionType(nil)
+ }
+
return b.m
}
type manifestFileV2 struct {
+ partitionType *StructType
Review Comment:
Could this be integrated with the `FieldSummary` instead of being at the top level here?
##########
internal/manifest_list_v2.go:
##########
@@ -0,0 +1,100 @@
+package internal
+
+import (
+ "github.com/hamba/avro/v2"
+)
+
+func MustNewManifestListV2Schema() avro.Schema {
Review Comment:
since this is a static schema, can we cache it rather than building it on demand?
##########
internal/manifest-entry-v1.json:
##########
@@ -0,0 +1,182 @@
+{
Review Comment:
where does this json file get used?
##########
utils.go:
##########
@@ -238,3 +239,68 @@ func avroEncode[T any](key string, version int, vals []T, out io.Writer) error {
return enc.Close()
}
+
+func structTypeToAvroPartitionSchema(st *StructType) (avro.Schema, error) {
+ var aFields []*avro.Field
+
+ for _, field := range st.Fields() {
+ aField, err := nestedFieldToAvroField(field)
+
+ if err != nil {
+ return nil, err
+ }
+
+ aFields = append(aFields, aField)
+ }
+
+ return avro.NewRecordSchema("r102", "", aFields)
Review Comment:
should this be "r102"? Or should this be more dynamic?
##########
internal/manifest_list_v1.go:
##########
@@ -0,0 +1,142 @@
+package internal
+
+import (
+ "github.com/hamba/avro/v2"
+)
+
+func MustNewManifestListV1Schema() avro.Schema {
Review Comment:
same as other comments
##########
manifest.go:
##########
@@ -613,31 +633,37 @@ func WriteManifestList(out io.Writer, files []ManifestFile) error {
}
}
- var key string
+ var sch avro.Schema
switch version {
case 1:
- key = internal.ManifestListV1Key
+ sch = internal.MustNewManifestListV1Schema()
case 2:
- key = internal.ManifestListV2Key
+ sch = internal.MustNewManifestListV2Schema()
default:
return fmt.Errorf("%w: non-recognized version %d", ErrInvalidArgument, version)
}
Review Comment:
for the ManifestList schema and others that don't need the dynamic part, it's better to have a cache of the schemas than to build it from scratch each time.
##########
internal/utils.go:
##########
@@ -0,0 +1,35 @@
+package internal
+
+import (
+ "github.com/hamba/avro/v2"
+)
+
+func MustNewUnionSchema(types []avro.Schema, opts ...avro.SchemaOption) *avro.UnionSchema {
+ s, err := avro.NewUnionSchema(types, opts...)
+
+ if err != nil {
+ panic(err)
+ }
+
+ return s
+}
Review Comment:
these can all be simplified with a simple
```go
func Must[T any](v T, err error) T {
if err != nil {
panic(err)
}
return v
}
```
and then just using that generic method around the rest of the calls instead. Honestly, we probably can just have that utility function and not bother having these at all, just calling `Must(avro.NewUnionSchema(.....))` etc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Fix #305: ManifestEntry partition field schema should be dynamically … [iceberg-go]
Posted by "arnaudbriche (via GitHub)" <gi...@apache.org>.
arnaudbriche commented on PR #307:
URL: https://github.com/apache/iceberg-go/pull/307#issuecomment-2671135678
I'm trying to create and maintain Iceberg tables from externally generated Parquet files with this lib.
The goal is for Clickhouse to be able to query the table.
While writing the various Avro metadata files with the lib, I encountered many errors from ClickHouse, which complained about missing metadata on the Avro manifest files.
As of the spec, Manifest files must have specific metadata to be valid: https://iceberg.apache.org/spec/#manifests
I thus created the `WriteManifestEntries` that writes the Avro Manifest files with proper metadata.
Not sure how to make the `WriteEntries` method of `manifestFileV1` and `manifestFileV2` write compliant Avro Manifest files.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Fix #305: ManifestEntry partition field schema should be dynamically … [iceberg-go]
Posted by "arnaudbriche (via GitHub)" <gi...@apache.org>.
arnaudbriche commented on PR #307:
URL: https://github.com/apache/iceberg-go/pull/307#issuecomment-2675831964
@zeroshade it's working fine !
API looks better.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Fix #305: ManifestEntry partition field schema should be dynamically … [iceberg-go]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on PR #307:
URL: https://github.com/apache/iceberg-go/pull/307#issuecomment-2675721589
@arnaudbriche would it be easier for you to enable maintainers to push to this branch and update or should I just open a new PR with the changes?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Fix #305: ManifestEntry partition field schema should be dynamically … [iceberg-go]
Posted by "arnaudbriche (via GitHub)" <gi...@apache.org>.
arnaudbriche commented on PR #307:
URL: https://github.com/apache/iceberg-go/pull/307#issuecomment-2675839884
Nothing to do with the PR but just a quick question regarding io API.
Why is there a requirement for `io.ReadSeekCloser` on `File` ? It seems like `Seek` is not used anywhere, and having to implement it to be compliant with `IO` is a bit annoying.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Fix #305: ManifestEntry partition field schema should be dynamically … [iceberg-go]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on PR #307:
URL: https://github.com/apache/iceberg-go/pull/307#issuecomment-2675783109
Looks like i have read access but not write access currently :frowning:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] fix(manifest): ManifestEntry partition field schema should be dynamically generated [iceberg-go]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade merged PR #307:
URL: https://github.com/apache/iceberg-go/pull/307
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org