You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "valkum (via GitHub)" <gi...@apache.org> on 2025/09/22 12:33:10 UTC
[I] Column with List(Struct) causes failed to decode level data for struct array [arrow-rs]
valkum opened a new issue, #8404:
URL: https://github.com/apache/arrow-rs/issues/8404
**Describe the bug**
<!--
A clear and concise description of what the bug is.
-->
I have a file generated using polars which polars, duckdb and other tools read fine. arrow-rs fails with a `Failed to decode level data for struct array` error.
**To Reproduce**
<!--
Steps to reproduce the behavior:
-->
Try to read the following file: https://storage.googleapis.com/ids-next/arrow-bug-dremel-encoding.parquet e.g. via
```
pub(crate) fn bug() -> () {
let src = "https://storage.googleapis.com/ids-next/arrow-bug-dremel-encoding.parquet";
let mut reader = File::open(src.strip_prefix("file://").unwrap()).unwrap();
let metadata = ArrowReaderMetadata::load(&mut reader, ArrowReaderOptions::default()).unwrap();
let mut market_reader =
ParquetRecordBatchReaderBuilder::new_with_metadata(reader, metadata.clone())
.build()
.unwrap();
let mut count = 0;
while let Some(batch) = market_reader.next() {
let batch = batch.unwrap();
count += batch.num_rows();
}
println!("Read {} rows", count);
()
}
```
**Expected behavior**
<!--
A clear and concise description of what you expected to happen.
-->
The file should read or provide a better error if the file contains something unsupported.
**Additional context**
<!--
Add any other context about the problem here.
-->
I tried to come up with a unit test for struct_array.rs but ran out of time.
I tried this with a struct of only ints, but that didn't trigger the bug; thus, I assume the dict might be part of the cause.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Column with List(Struct) causes failed to decode level data for struct array [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #8404:
URL: https://github.com/apache/arrow-rs/issues/8404#issuecomment-3378495564
Thanks @valkum. It looks like https://github.com/apache/arrow-rs/commit/e814b97415d2df78c273302dab934dfe2a0a4a64 was part of this PR:
- https://github.com/apache/arrow-rs/pull/7585
I'll comment there and see if we can get some more attention on this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Column with List(Struct) causes failed to decode level data for struct array [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #8404:
URL: https://github.com/apache/arrow-rs/issues/8404#issuecomment-3353729281
I tested this file with the PR from @rluvaton in
- https://github.com/apache/arrow-rs/pull/8496
And unfortunately, it still fails:
```
called `Result::unwrap()` on an `Err` value: ParquetError("Parquet error: Failed to decode level data for struct array")
thread 'test_bug' panicked at parquet/tests/arrow_reader/mod.rs:1171:27:
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Column with List(Struct) causes failed to decode level data for struct array [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #8404:
URL: https://github.com/apache/arrow-rs/issues/8404#issuecomment-3330775763
Here is the reproducer:
[arrow-bug-dremel-encoding.parquet.zip](https://github.com/user-attachments/files/22523297/arrow-bug-dremel-encoding.parquet.zip)
DataFusion CLI indeed has an error:
```shell
DataFusion CLI v50.0.0
> select * from 'arrow-bug-dremel-encoding.parquet';
Parquet error: Arrow: Parquet argument error: Parquet error: Failed to decode level data for struct array
```
DuckDB can read this file
```
D select * from read_parquet('arrow-bug-dremel-encoding.parquet');
┌─────────────┬─────────┬───────────┬─────────────┬─────────┬─────────────────┬──────────────────────┬────────────┬─────────────────────────────────────────────────────┐
│ registrable │ etld │ is_market │ is_expiring │ zone │ security_trails │ markets │ domain │ embedding │
│ varchar │ varchar │ boolean │ boolean │ boolean │ boolean │ struct(market varc… │ varchar │ float[] │
├─────────────┼─────────┼───────────┼─────────────┼─────────┼─────────────────┼──────────────────────┼────────────┼─────────────────────────────────────────────────────┤
│ en13549 │ eu │ true │ false │ false │ true │ [{'market': sedo.c… │ en13549.eu │ [-0.056253448, -0.026599279, -0.05077901, -0.0911… │
└─────────────┴─────────┴───────────┴─────────────┴─────────┴─────────────────┴──────────────────────┴────────────┴─────────────────────────────────────────────────────┘
D
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Column with List(Struct) causes failed to decode level data for struct array [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #8404:
URL: https://github.com/apache/arrow-rs/issues/8404#issuecomment-3416676764
`label_issue.py` automatically added labels {'parquet'} from #8573
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Column with List(Struct) causes failed to decode level data for struct array [arrow-rs]
Posted by "valkum (via GitHub)" <gi...@apache.org>.
valkum commented on issue #8404:
URL: https://github.com/apache/arrow-rs/issues/8404#issuecomment-3355736883
The schema looks correct from what I can tell. I tried to write some debugging tools for this case the past couple of days if I got a minute, but I wasn't able to find eht root cuase yet.
Here is the schema for reference:
<details><summary># parquet-tools schema --format raw ~/Downloads/arrow-bug-dremel-encoding.parquet |jq</summary>
<p>
```
"name": "root",
"num_children": 9,
"children": [
{
"type": "BYTE_ARRAY",
"repetition_type": "OPTIONAL",
"name": "registrable",
"converted_type": "UTF8",
"logicalType": {
"STRING": {}
}
},
{
"type": "BYTE_ARRAY",
"repetition_type": "OPTIONAL",
"name": "etld",
"converted_type": "UTF8",
"logicalType": {
"STRING": {}
}
},
{
"type": "BOOLEAN",
"repetition_type": "OPTIONAL",
"name": "is_market"
},
{
"type": "BOOLEAN",
"repetition_type": "OPTIONAL",
"name": "is_expiring"
},
{
"type": "BOOLEAN",
"repetition_type": "OPTIONAL",
"name": "zone"
},
{
"type": "BOOLEAN",
"repetition_type": "OPTIONAL",
"name": "security_trails"
},
{
"repetition_type": "OPTIONAL",
"name": "markets",
"num_children": 1,
"converted_type": "LIST",
"logicalType": {
"LIST": {}
},
"children": [
{
"repetition_type": "REPEATED",
"name": "list",
"num_children": 1,
"children": [
{
"repetition_type": "OPTIONAL",
"name": "element",
"num_children": 6,
"children": [
{
"type": "BYTE_ARRAY",
"repetition_type": "OPTIONAL",
"name": "market",
"converted_type": "UTF8",
"logicalType": {
"STRING": {}
}
},
{
"type": "INT64",
"repetition_type": "OPTIONAL",
"name": "expiring",
"logicalType": {
"TIMESTAMP": {
"isAdjustedToUTC": false,
"unit": {
"MILLIS": {}
}
}
}
},
{
"type": "INT64",
"repetition_type": "OPTIONAL",
"name": "price",
"converted_type": "UINT_64",
"logicalType": {
"INTEGER": {
"bitWidth": 64,
"isSigned": false
}
}
},
{
"type": "INT64",
"repetition_type": "OPTIONAL",
"name": "min_price",
"converted_type": "UINT_64",
"logicalType": {
"INTEGER": {
"bitWidth": 64,
"isSigned": false
}
}
},
{
"type": "INT64",
"repetition_type": "OPTIONAL",
"name": "valuation",
"converted_type": "UINT_64",
"logicalType": {
"INTEGER": {
"bitWidth": 64,
"isSigned": false
}
}
},
{
"type": "BYTE_ARRAY",
"repetition_type": "OPTIONAL",
"name": "type",
"converted_type": "UTF8",
"logicalType": {
"STRING": {}
}
}
]
}
]
}
]
},
{
"type": "BYTE_ARRAY",
"repetition_type": "OPTIONAL",
"name": "domain",
"converted_type": "UTF8",
"logicalType": {
"STRING": {}
}
},
{
"repetition_type": "OPTIONAL",
"name": "embedding",
"num_children": 1,
"converted_type": "LIST",
"logicalType": {
"LIST": {}
},
"children": [
{
"repetition_type": "REPEATED",
"name": "list",
"num_children": 1,
"children": [
{
"type": "FLOAT",
"repetition_type": "OPTIONAL",
"name": "element"
}
]
}
]
}
]
}
```
</p>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Column with List(Struct) causes failed to decode level data for struct array [arrow-rs]
Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl closed issue #8404: Column with List(Struct) causes failed to decode level data for struct array
URL: https://github.com/apache/arrow-rs/issues/8404
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Column with List(Struct) causes failed to decode level data for struct array [arrow-rs]
Posted by "valkum (via GitHub)" <gi...@apache.org>.
valkum commented on issue #8404:
URL: https://github.com/apache/arrow-rs/issues/8404#issuecomment-3378439532
I had a bit of time to spend on this again and figured out that this was actually a regression introduced by 55.2.0. Bisecting showed that e814b97415d2df78c273302dab934dfe2a0a4a64 regressed.
Both the market and the type leaf columns report 0 records and values in the `GenericRecordReader`.
I see two calls to `ByteArrayDictionaryReader::consume_batch` for each column (etld, type, market); in the second call the values are already taken by `GenericRecordReader::consume_record_data`
One note: It looks like Polars created a shared `LargeStringArray` Dict for all cols (etld, type, market).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Column with List(Struct) causes failed to decode level data for struct array [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #8404:
URL: https://github.com/apache/arrow-rs/issues/8404#issuecomment-3380829913
@albertlockett has graciously offered to look into this issue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org