You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "zeroshade (via GitHub)" <gi...@apache.org> on 2025/04/28 15:17:00 UTC
[PR] fix(table): Handle nullable struct with Required Field [iceberg-go]
zeroshade opened a new pull request, #408:
URL: https://github.com/apache/iceberg-go/pull/408
fixes #398
Have `toRequestedSchema` use the new `NewStructArrayWithFieldsAndNulls` function to correctly propagate null info and handle required fields within a nullable struct.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] fix(table): Handle nullable struct with Required Field [iceberg-go]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on code in PR #408:
URL: https://github.com/apache/iceberg-go/pull/408#discussion_r2069102519
##########
table/arrow_utils.go:
##########
@@ -417,6 +418,15 @@ func ArrowSchemaToIceberg(sc *arrow.Schema, downcastNsTimestamp bool, nameMappin
}
}
+func ArrowSchemaToIcebergWithoutIDs(sc *arrow.Schema, downcastNsTimestamp bool) (*iceberg.Schema, error) {
Review Comment:
I think this name is confusing since we add fresh-IDs?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] fix(table): Handle nullable struct with Required Field [iceberg-go]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #408:
URL: https://github.com/apache/iceberg-go/pull/408#discussion_r2066935738
##########
table/table_test.go:
##########
@@ -1135,3 +1136,73 @@ func TestTableWriting(t *testing.T) {
suite.Run(t, &TableWritingTestSuite{formatVersion: 1})
suite.Run(t, &TableWritingTestSuite{formatVersion: 2})
}
+
+func TestNullableStructRequiredField(t *testing.T) {
+ loc := t.TempDir()
+
+ cat, err := catalog.Load(context.Background(), "default", iceberg.Properties{
+ "uri": ":memory:",
+ "type": "sql",
+ sql.DriverKey: sqliteshim.ShimName,
+ sql.DialectKey: string(sql.SQLite),
+ "warehouse": "file://" + loc,
+ })
+ require.NoError(t, err)
+
+ arrowSchema := arrow.NewSchema([]arrow.Field{
+ {
+ Name: "analytic", Type: arrow.StructOf(
+ arrow.Field{Name: "category", Type: arrow.BinaryTypes.String, Nullable: true},
+ arrow.Field{Name: "desc", Type: arrow.BinaryTypes.String, Nullable: true},
+ arrow.Field{Name: "name", Type: arrow.BinaryTypes.String, Nullable: true},
+ arrow.Field{Name: "related_analytics", Type: arrow.ListOf(
+ arrow.StructOf(
+ arrow.Field{Name: "category", Type: arrow.BinaryTypes.String, Nullable: true},
+ arrow.Field{Name: "desc", Type: arrow.BinaryTypes.String, Nullable: true},
+ arrow.Field{Name: "name", Type: arrow.BinaryTypes.String, Nullable: true},
+ arrow.Field{Name: "type", Type: arrow.BinaryTypes.String, Nullable: true},
+ arrow.Field{Name: "type_id", Type: arrow.PrimitiveTypes.Int32, Nullable: false},
+ arrow.Field{Name: "uid", Type: arrow.BinaryTypes.String, Nullable: true},
+ arrow.Field{Name: "version", Type: arrow.BinaryTypes.String, Nullable: true},
+ ),
+ ), Nullable: true},
+ arrow.Field{Name: "type", Type: arrow.BinaryTypes.String, Nullable: true},
+ arrow.Field{Name: "type_id", Type: arrow.PrimitiveTypes.Int32, Nullable: false},
+ arrow.Field{Name: "uid", Type: arrow.BinaryTypes.String, Nullable: true},
+ arrow.Field{Name: "version", Type: arrow.BinaryTypes.String, Nullable: true},
+ ), Nullable: true,
+ },
+ {Name: "uid", Type: arrow.BinaryTypes.String, Nullable: true},
+ }, nil)
+
+ sc, err := table.ArrowSchemaToIcebergWithoutIDs(arrowSchema, false)
+ if err != nil {
+ panic(err)
Review Comment:
nope, just an artifact from a previous version. Good catch, I'll fix that
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] fix(table): Handle nullable struct with Required Field [iceberg-go]
Posted by "pdelewski (via GitHub)" <gi...@apache.org>.
pdelewski commented on code in PR #408:
URL: https://github.com/apache/iceberg-go/pull/408#discussion_r2068335628
##########
table/table_test.go:
##########
@@ -1135,3 +1136,71 @@ func TestTableWriting(t *testing.T) {
suite.Run(t, &TableWritingTestSuite{formatVersion: 1})
suite.Run(t, &TableWritingTestSuite{formatVersion: 2})
}
+
+func TestNullableStructRequiredField(t *testing.T) {
+ loc := t.TempDir()
+
+ cat, err := catalog.Load(context.Background(), "default", iceberg.Properties{
+ "uri": ":memory:",
+ "type": "sql",
+ sql.DriverKey: sqliteshim.ShimName,
+ sql.DialectKey: string(sql.SQLite),
+ "warehouse": "file://" + loc,
+ })
+ require.NoError(t, err)
+
+ arrowSchema := arrow.NewSchema([]arrow.Field{
+ {
+ Name: "analytic", Type: arrow.StructOf(
+ arrow.Field{Name: "category", Type: arrow.BinaryTypes.String, Nullable: true},
+ arrow.Field{Name: "desc", Type: arrow.BinaryTypes.String, Nullable: true},
+ arrow.Field{Name: "name", Type: arrow.BinaryTypes.String, Nullable: true},
+ arrow.Field{Name: "related_analytics", Type: arrow.ListOf(
+ arrow.StructOf(
+ arrow.Field{Name: "category", Type: arrow.BinaryTypes.String, Nullable: true},
+ arrow.Field{Name: "desc", Type: arrow.BinaryTypes.String, Nullable: true},
+ arrow.Field{Name: "name", Type: arrow.BinaryTypes.String, Nullable: true},
+ arrow.Field{Name: "type", Type: arrow.BinaryTypes.String, Nullable: true},
+ arrow.Field{Name: "type_id", Type: arrow.PrimitiveTypes.Int32, Nullable: false},
+ arrow.Field{Name: "uid", Type: arrow.BinaryTypes.String, Nullable: true},
+ arrow.Field{Name: "version", Type: arrow.BinaryTypes.String, Nullable: true},
+ ),
+ ), Nullable: true},
+ arrow.Field{Name: "type", Type: arrow.BinaryTypes.String, Nullable: true},
+ arrow.Field{Name: "type_id", Type: arrow.PrimitiveTypes.Int32, Nullable: false},
+ arrow.Field{Name: "uid", Type: arrow.BinaryTypes.String, Nullable: true},
+ arrow.Field{Name: "version", Type: arrow.BinaryTypes.String, Nullable: true},
+ ), Nullable: true,
+ },
+ {Name: "uid", Type: arrow.BinaryTypes.String, Nullable: true},
+ }, nil)
+
+ sc, err := table.ArrowSchemaToIcebergWithoutIDs(arrowSchema, false)
+ require.NoError(t, err)
+
+ ctx := context.TODO()
+ require.NoError(t, cat.CreateNamespace(ctx, table.Identifier{"testing"}, nil))
+ tbl, err := cat.CreateTable(ctx, table.Identifier{"testing", "nullable_struct_required_field"}, sc,
+ catalog.WithProperties(iceberg.Properties{"format-version": "2"}),
+ catalog.WithLocation(loc))
+ require.NoError(t, err)
+ require.NotNil(t, tbl)
+
+ bldr := array.NewRecordBuilder(memory.DefaultAllocator, arrowSchema)
+ defer bldr.Release()
+
+ bldr.Field(0).AppendNulls(100)
Review Comment:
nitpick: I think this `100` could be extracted as const as it's repeated 3 times
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] fix(table): Handle nullable struct with Required Field [iceberg-go]
Posted by "EthanBlackburn (via GitHub)" <gi...@apache.org>.
EthanBlackburn commented on PR #408:
URL: https://github.com/apache/iceberg-go/pull/408#issuecomment-2837261873
@zeroshade it works! 🥳
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] fix(table): Handle nullable struct with Required Field [iceberg-go]
Posted by "pdelewski (via GitHub)" <gi...@apache.org>.
pdelewski commented on code in PR #408:
URL: https://github.com/apache/iceberg-go/pull/408#discussion_r2066810271
##########
table/arrow_utils.go:
##########
@@ -813,7 +823,26 @@ func (a *arrowProjectionVisitor) Struct(st iceberg.StructType, structArr arrow.A
}
}
- return retOrPanic(array.NewStructArrayWithFields(fieldArrs, fields))
+ var nullBitmap *memory.Buffer
+ if structArr.NullN() > 0 {
Review Comment:
Would not be cleaner to reverse condition and return
```suggestion
if structArr.NullN() <= 0 {
return retOrPanic(array.NewStructArrayWithFieldsAndNulls(fieldArrs, fields,
nullBitmap, structArr.NullN(), 0))
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] fix(table): Handle nullable struct with Required Field [iceberg-go]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #408:
URL: https://github.com/apache/iceberg-go/pull/408#discussion_r2066934954
##########
table/arrow_utils.go:
##########
@@ -813,7 +823,26 @@ func (a *arrowProjectionVisitor) Struct(st iceberg.StructType, structArr arrow.A
}
}
- return retOrPanic(array.NewStructArrayWithFields(fieldArrs, fields))
+ var nullBitmap *memory.Buffer
+ if structArr.NullN() > 0 {
Review Comment:
I don't think that would make it any cleaner really. Unless I'm missing something, all that would do is force us to duplicate the return line and only remove one level of braces.
Am I misunderstanding?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] fix(table): Handle nullable struct with Required Field [iceberg-go]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on code in PR #408:
URL: https://github.com/apache/iceberg-go/pull/408#discussion_r2069103524
##########
table/arrow_utils.go:
##########
@@ -417,6 +418,15 @@ func ArrowSchemaToIceberg(sc *arrow.Schema, downcastNsTimestamp bool, nameMappin
}
}
+func ArrowSchemaToIcebergWithoutIDs(sc *arrow.Schema, downcastNsTimestamp bool) (*iceberg.Schema, error) {
Review Comment:
What about:
```suggestion
func ArrowSchemaToIcebergWithFreshIDs(sc *arrow.Schema, downcastNsTimestamp bool) (*iceberg.Schema, error) {
```
Open for other suggestions
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] fix(table): Handle nullable struct with Required Field [iceberg-go]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on PR #408:
URL: https://github.com/apache/iceberg-go/pull/408#issuecomment-2835646605
@EthanBlackburn please take a look and try this out when you get a chance. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] fix(table): Handle nullable struct with Required Field [iceberg-go]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #408:
URL: https://github.com/apache/iceberg-go/pull/408#discussion_r2069156439
##########
table/arrow_utils.go:
##########
@@ -417,6 +418,15 @@ func ArrowSchemaToIceberg(sc *arrow.Schema, downcastNsTimestamp bool, nameMappin
}
}
+func ArrowSchemaToIcebergWithoutIDs(sc *arrow.Schema, downcastNsTimestamp bool) (*iceberg.Schema, error) {
Review Comment:
talked with Fokko separately and we decided we'll keep this in for now (albeit renamed). and remove/rework it in the future if necessary.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] fix(table): Handle nullable struct with Required Field [iceberg-go]
Posted by "pdelewski (via GitHub)" <gi...@apache.org>.
pdelewski commented on code in PR #408:
URL: https://github.com/apache/iceberg-go/pull/408#discussion_r2066816991
##########
table/table_test.go:
##########
@@ -1135,3 +1136,73 @@ func TestTableWriting(t *testing.T) {
suite.Run(t, &TableWritingTestSuite{formatVersion: 1})
suite.Run(t, &TableWritingTestSuite{formatVersion: 2})
}
+
+func TestNullableStructRequiredField(t *testing.T) {
+ loc := t.TempDir()
+
+ cat, err := catalog.Load(context.Background(), "default", iceberg.Properties{
+ "uri": ":memory:",
+ "type": "sql",
+ sql.DriverKey: sqliteshim.ShimName,
+ sql.DialectKey: string(sql.SQLite),
+ "warehouse": "file://" + loc,
+ })
+ require.NoError(t, err)
+
+ arrowSchema := arrow.NewSchema([]arrow.Field{
+ {
+ Name: "analytic", Type: arrow.StructOf(
+ arrow.Field{Name: "category", Type: arrow.BinaryTypes.String, Nullable: true},
+ arrow.Field{Name: "desc", Type: arrow.BinaryTypes.String, Nullable: true},
+ arrow.Field{Name: "name", Type: arrow.BinaryTypes.String, Nullable: true},
+ arrow.Field{Name: "related_analytics", Type: arrow.ListOf(
+ arrow.StructOf(
+ arrow.Field{Name: "category", Type: arrow.BinaryTypes.String, Nullable: true},
+ arrow.Field{Name: "desc", Type: arrow.BinaryTypes.String, Nullable: true},
+ arrow.Field{Name: "name", Type: arrow.BinaryTypes.String, Nullable: true},
+ arrow.Field{Name: "type", Type: arrow.BinaryTypes.String, Nullable: true},
+ arrow.Field{Name: "type_id", Type: arrow.PrimitiveTypes.Int32, Nullable: false},
+ arrow.Field{Name: "uid", Type: arrow.BinaryTypes.String, Nullable: true},
+ arrow.Field{Name: "version", Type: arrow.BinaryTypes.String, Nullable: true},
+ ),
+ ), Nullable: true},
+ arrow.Field{Name: "type", Type: arrow.BinaryTypes.String, Nullable: true},
+ arrow.Field{Name: "type_id", Type: arrow.PrimitiveTypes.Int32, Nullable: false},
+ arrow.Field{Name: "uid", Type: arrow.BinaryTypes.String, Nullable: true},
+ arrow.Field{Name: "version", Type: arrow.BinaryTypes.String, Nullable: true},
+ ), Nullable: true,
+ },
+ {Name: "uid", Type: arrow.BinaryTypes.String, Nullable: true},
+ }, nil)
+
+ sc, err := table.ArrowSchemaToIcebergWithoutIDs(arrowSchema, false)
+ if err != nil {
+ panic(err)
Review Comment:
is any reason to use `panic` here instead of `require.NoError(t, err)`
##########
table/table_test.go:
##########
@@ -1135,3 +1136,73 @@ func TestTableWriting(t *testing.T) {
suite.Run(t, &TableWritingTestSuite{formatVersion: 1})
suite.Run(t, &TableWritingTestSuite{formatVersion: 2})
}
+
+func TestNullableStructRequiredField(t *testing.T) {
+ loc := t.TempDir()
+
+ cat, err := catalog.Load(context.Background(), "default", iceberg.Properties{
+ "uri": ":memory:",
+ "type": "sql",
+ sql.DriverKey: sqliteshim.ShimName,
+ sql.DialectKey: string(sql.SQLite),
+ "warehouse": "file://" + loc,
+ })
+ require.NoError(t, err)
+
+ arrowSchema := arrow.NewSchema([]arrow.Field{
+ {
+ Name: "analytic", Type: arrow.StructOf(
+ arrow.Field{Name: "category", Type: arrow.BinaryTypes.String, Nullable: true},
+ arrow.Field{Name: "desc", Type: arrow.BinaryTypes.String, Nullable: true},
+ arrow.Field{Name: "name", Type: arrow.BinaryTypes.String, Nullable: true},
+ arrow.Field{Name: "related_analytics", Type: arrow.ListOf(
+ arrow.StructOf(
+ arrow.Field{Name: "category", Type: arrow.BinaryTypes.String, Nullable: true},
+ arrow.Field{Name: "desc", Type: arrow.BinaryTypes.String, Nullable: true},
+ arrow.Field{Name: "name", Type: arrow.BinaryTypes.String, Nullable: true},
+ arrow.Field{Name: "type", Type: arrow.BinaryTypes.String, Nullable: true},
+ arrow.Field{Name: "type_id", Type: arrow.PrimitiveTypes.Int32, Nullable: false},
+ arrow.Field{Name: "uid", Type: arrow.BinaryTypes.String, Nullable: true},
+ arrow.Field{Name: "version", Type: arrow.BinaryTypes.String, Nullable: true},
+ ),
+ ), Nullable: true},
+ arrow.Field{Name: "type", Type: arrow.BinaryTypes.String, Nullable: true},
+ arrow.Field{Name: "type_id", Type: arrow.PrimitiveTypes.Int32, Nullable: false},
+ arrow.Field{Name: "uid", Type: arrow.BinaryTypes.String, Nullable: true},
+ arrow.Field{Name: "version", Type: arrow.BinaryTypes.String, Nullable: true},
+ ), Nullable: true,
+ },
+ {Name: "uid", Type: arrow.BinaryTypes.String, Nullable: true},
+ }, nil)
+
+ sc, err := table.ArrowSchemaToIcebergWithoutIDs(arrowSchema, false)
+ if err != nil {
+ panic(err)
Review Comment:
is any reason to use `panic` here instead of `require.NoError(t, err)`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] fix(table): Handle nullable struct with Required Field [iceberg-go]
Posted by "pdelewski (via GitHub)" <gi...@apache.org>.
pdelewski commented on code in PR #408:
URL: https://github.com/apache/iceberg-go/pull/408#discussion_r2068285429
##########
table/arrow_utils.go:
##########
@@ -813,7 +823,26 @@ func (a *arrowProjectionVisitor) Struct(st iceberg.StructType, structArr arrow.A
}
}
- return retOrPanic(array.NewStructArrayWithFields(fieldArrs, fields))
+ var nullBitmap *memory.Buffer
+ if structArr.NullN() > 0 {
Review Comment:
You're right — I analyzed this logic incorrectly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] fix(table): Handle nullable struct with Required Field [iceberg-go]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #408:
URL: https://github.com/apache/iceberg-go/pull/408#discussion_r2069125746
##########
table/table_test.go:
##########
@@ -1135,3 +1136,71 @@ func TestTableWriting(t *testing.T) {
suite.Run(t, &TableWritingTestSuite{formatVersion: 1})
suite.Run(t, &TableWritingTestSuite{formatVersion: 2})
}
+
+func TestNullableStructRequiredField(t *testing.T) {
+ loc := t.TempDir()
+
+ cat, err := catalog.Load(context.Background(), "default", iceberg.Properties{
+ "uri": ":memory:",
+ "type": "sql",
+ sql.DriverKey: sqliteshim.ShimName,
+ sql.DialectKey: string(sql.SQLite),
+ "warehouse": "file://" + loc,
+ })
+ require.NoError(t, err)
+
+ arrowSchema := arrow.NewSchema([]arrow.Field{
+ {
+ Name: "analytic", Type: arrow.StructOf(
+ arrow.Field{Name: "category", Type: arrow.BinaryTypes.String, Nullable: true},
+ arrow.Field{Name: "desc", Type: arrow.BinaryTypes.String, Nullable: true},
+ arrow.Field{Name: "name", Type: arrow.BinaryTypes.String, Nullable: true},
+ arrow.Field{Name: "related_analytics", Type: arrow.ListOf(
+ arrow.StructOf(
+ arrow.Field{Name: "category", Type: arrow.BinaryTypes.String, Nullable: true},
+ arrow.Field{Name: "desc", Type: arrow.BinaryTypes.String, Nullable: true},
+ arrow.Field{Name: "name", Type: arrow.BinaryTypes.String, Nullable: true},
+ arrow.Field{Name: "type", Type: arrow.BinaryTypes.String, Nullable: true},
+ arrow.Field{Name: "type_id", Type: arrow.PrimitiveTypes.Int32, Nullable: false},
+ arrow.Field{Name: "uid", Type: arrow.BinaryTypes.String, Nullable: true},
+ arrow.Field{Name: "version", Type: arrow.BinaryTypes.String, Nullable: true},
+ ),
+ ), Nullable: true},
+ arrow.Field{Name: "type", Type: arrow.BinaryTypes.String, Nullable: true},
+ arrow.Field{Name: "type_id", Type: arrow.PrimitiveTypes.Int32, Nullable: false},
+ arrow.Field{Name: "uid", Type: arrow.BinaryTypes.String, Nullable: true},
+ arrow.Field{Name: "version", Type: arrow.BinaryTypes.String, Nullable: true},
+ ), Nullable: true,
+ },
+ {Name: "uid", Type: arrow.BinaryTypes.String, Nullable: true},
+ }, nil)
+
+ sc, err := table.ArrowSchemaToIcebergWithoutIDs(arrowSchema, false)
+ require.NoError(t, err)
+
+ ctx := context.TODO()
+ require.NoError(t, cat.CreateNamespace(ctx, table.Identifier{"testing"}, nil))
+ tbl, err := cat.CreateTable(ctx, table.Identifier{"testing", "nullable_struct_required_field"}, sc,
+ catalog.WithProperties(iceberg.Properties{"format-version": "2"}),
+ catalog.WithLocation(loc))
+ require.NoError(t, err)
+ require.NotNil(t, tbl)
+
+ bldr := array.NewRecordBuilder(memory.DefaultAllocator, arrowSchema)
+ defer bldr.Release()
+
+ bldr.Field(0).AppendNulls(100)
Review Comment:
sure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] fix(table): Handle nullable struct with Required Field [iceberg-go]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade merged PR #408:
URL: https://github.com/apache/iceberg-go/pull/408
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] fix(table): Handle nullable struct with Required Field [iceberg-go]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on code in PR #408:
URL: https://github.com/apache/iceberg-go/pull/408#discussion_r2069105644
##########
table/arrow_utils.go:
##########
@@ -417,6 +418,15 @@ func ArrowSchemaToIceberg(sc *arrow.Schema, downcastNsTimestamp bool, nameMappin
}
}
+func ArrowSchemaToIcebergWithoutIDs(sc *arrow.Schema, downcastNsTimestamp bool) (*iceberg.Schema, error) {
Review Comment:
What about moving this to testing?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org