You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "EstherLCode (via GitHub)" <gi...@apache.org> on 2025/04/17 07:08:37 UTC
[I] Java api appendFile leads to null value of array/struct column [iceberg]
EstherLCode opened a new issue, #12825:
URL: https://github.com/apache/iceberg/issues/12825
### Apache Iceberg version
1.4.3
### Query engine
Spark
### Please describe the bug 🐞
Hi there,
I have a parquet file called `a.parquet` generated from a jso. The schema of `a.parquet` is:
`
message schema {
optional binary stringField (STRING);
optional group objectField {
optional binary innerString (STRING);
optional group innerArray (LIST) {
repeated group list {
optional binary element (STRING);
}
}
optional group innerObject {
optional binary nestedString (STRING);
optional binary nestedBoolean (STRING);
optional group nestedArray (LIST) {
repeated group list {
optional group element {
optional binary f1 (STRING);
optional binary f2 (STRING);
optional binary f4 (STRING);
}
}
}
}
}
optional group arrayField (LIST) {
repeated group list {
optional group element {
optional binary nestedString (STRING);
optional binary nestedNumber (STRING);
optional binary nestedBoolean (STRING);
}
}
}
}
`
Using this schema to create an iceberg table `db1.tb1`.
And the content of `a.parquet` is:
`
scala> spark.read.parquet("/baseDir/a.parquet").show(false)
+-----------+-----------------------------------------------------------------------------------------------+----------------------------------------------------+
|stringField|objectField |arrayField |
+-----------+-----------------------------------------------------------------------------------------------+----------------------------------------------------+
|value |{innerValue, [1, 2, 3], {v1, true, [{v1, null, null}, {null, 1.0, true}, {null, null, false}]}}|[{nestedValue, 42, null}, {nestedValue, null, true}]|
+-----------+-----------------------------------------------------------------------------------------------+----------------------------------------------------+
`
I tried to add this parquet to an iceberg table 'db1.tb1' by **appendFile** function
`
val table = catalog.loadTable(TableIdentifier.of("db1","tb1")
val dataFile = DataFiles.builder(table.spec())
.withPath("/baseDir/a.parquet")
.withFormat(FileFormat.PARQUET)
.withFileSizeInBytes(size)
.withRecordCount(spark.read.parquet("/baseDir/a.parquet").count)
.build()
table.newAppend()
.appendFile(dataFile)
.commit()
`
But when I query the table, I got **null** for columns objectField and arrayField, like:
`
scala> spark.sql("select * from db1.tb1").show(false)
+-----------+-----------------------------------------------------------------------------------------------+----------------------------------------------------+
|stringField|objectField |arrayField |
+-----------+-----------------------------------------------------------------------------------------------+----------------------------------------------------+
|value |null|null|
+-----------+-----------------------------------------------------------------------------------------------+----------------------------------------------------+
`
The weird thing is then I tried the sql api `add_files`, this command worked well and also fixed the old record.
`
spark.sql("""call spark_catalog.system.add_files (table => 'spark_catalog.db1.tb1',source_table => '`parquet`.`hdfs:///baseDir/a.parquet`')""")
`
After this command, I got two completed and correct records.
`
scala> spark.sql("select * from db1.tb1").show(false)
+-----------+-----------------------------------------------------------------------------------------------+----------------------------------------------------+
|stringField|objectField |arrayField |
+-----------+-----------------------------------------------------------------------------------------------+----------------------------------------------------+
|value |{innerValue, [1, 2, 3], {v1, true, [{v1, null, null}, {null, 1.0, true}, {null, null, false}]}}|[{nestedValue, 42, null}, {nestedValue, null, true}]|
|value |{innerValue, [1, 2, 3], {v1, true, [{v1, null, null}, {null, 1.0, true}, {null, null, false}]}}|[{nestedValue, 42, null}, {nestedValue, null, true}]|
+-----------+-----------------------------------------------------------------------------------------------+----------------------------------------------------+
`
I'm wondering if my calling of `appendFile` is correct? And why here are different results between `appendFile` and `add_files`?
Thanks a lot.
### Willingness to contribute
- [ ] I can contribute a fix for this bug independently
- [x] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Java api appendFile leads to null value of array/struct column [iceberg]
Posted by "xiaoxuandev (via GitHub)" <gi...@apache.org>.
xiaoxuandev commented on issue #12825:
URL: https://github.com/apache/iceberg/issues/12825#issuecomment-2822364050
Hi @EstherLCode, are you still seeing this issue on 1.8.1?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Java api appendFile leads to null value of array/struct column [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #12825:
URL: https://github.com/apache/iceberg/issues/12825#issuecomment-3420095076
This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Java api appendFile leads to null value of array/struct column [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #12825:
URL: https://github.com/apache/iceberg/issues/12825#issuecomment-3483154134
This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Java api appendFile leads to null value of array/struct column [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #12825: Java api appendFile leads to null value of array/struct column
URL: https://github.com/apache/iceberg/issues/12825
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org