You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "lewiszlw (via GitHub)" <gi...@apache.org> on 2025/10/20 09:01:56 UTC
[I] Parquet 56: encounter `error: item_reader def levels are None` when reading nested field with row filter [arrow-rs]
lewiszlw opened a new issue, #8657:
URL: https://github.com/apache/arrow-rs/issues/8657
**Describe the bug**
<!--
A clear and concise description of what the bug is.
-->
After upgrading parquet to 56, we encountered `Parquet error: item_reader def levels are None` when reading nested field with row filter.
**To Reproduce**
<!--
Steps to reproduce the behavior:
-->
```rust
[dependencies]
arrow = "56"
parquet = { version = "56", features = ["async"] }
tokio = { version = "1", features = ["full"] }
futures = "0.3"
use arrow::array::{BooleanArray, Float32Builder, Int32Array, ListBuilder};
use arrow::datatypes::{DataType, Field, Schema};
use arrow::record_batch::RecordBatch;
use futures::stream::StreamExt;
use parquet::arrow::arrow_reader::{ArrowPredicateFn, RowFilter};
use parquet::arrow::{AsyncArrowWriter, ParquetRecordBatchStreamBuilder, ProjectionMask};
use std::sync::Arc;
use tokio::fs::File;
#[tokio::main]
async fn main() {
write_parquet_file().await.unwrap();
read_parquet_file().await.unwrap();
}
async fn read_parquet_file() -> Result<(), Box<dyn std::error::Error>> {
let file = File::open("output.parquet").await?;
let builder = ParquetRecordBatchStreamBuilder::new(file).await?;
let predicate = ArrowPredicateFn::new(ProjectionMask::all(), |batch| {
Ok(BooleanArray::from(vec![true; batch.num_rows()]))
});
let projection_mask = ProjectionMask::all();
let mut stream = builder
.with_row_filter(RowFilter::new(vec![Box::new(predicate)]))
.with_projection(projection_mask)
.build()?;
while let Some(batch) = stream.next().await {
let batch = batch?;
println!("Read batch {batch:?}");
}
Ok(())
}
async fn write_parquet_file() -> Result<(), Box<dyn std::error::Error>> {
let list_inner_field = Arc::new(Field::new("item", DataType::Float32, true));
let table_schema = Arc::new(Schema::new(vec![
Field::new("id", DataType::Int32, false),
Field::new("vector", DataType::List(list_inner_field.clone()), true),
]));
let mut list_builder =
ListBuilder::new(Float32Builder::new()).with_field(list_inner_field.clone());
list_builder.values().append_slice(&[10.0, 10.0, 10.0]);
list_builder.append(true);
list_builder.values().append_slice(&[20.0, 20.0, 20.0]);
list_builder.append(true);
list_builder.values().append_slice(&[30.0, 30.0, 30.0]);
list_builder.append(true);
list_builder.values().append_slice(&[40.0, 40.0, 40.0]);
list_builder.append(true);
let list_array = list_builder.finish();
let data = vec![RecordBatch::try_new(
table_schema.clone(),
vec![
Arc::new(Int32Array::from(vec![1, 2, 3, 4])),
Arc::new(list_array),
],
)?];
let file = File::create("output.parquet").await?;
let mut writer = AsyncArrowWriter::try_new(file, table_schema, None)?;
for batch in data {
writer.write(&batch).await?;
}
writer.close().await?;
println!("Parquet file written successfully!");
Ok(())
}
thread 'main' panicked at src\main.rs:13:31:
called `Result::unwrap()` on an `Err` value: External(ParquetError("Parquet error: item_reader def levels are None."))
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
```
Above code works in parquet 55.
**Expected behavior**
<!--
A clear and concise description of what you expected to happen.
-->
**Additional context**
<!--
Add any other context about the problem here.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Parquet 56: encounter `error: item_reader def levels are None` when reading nested field with row filter [arrow-rs]
Posted by "lewiszlw (via GitHub)" <gi...@apache.org>.
lewiszlw commented on issue #8657:
URL: https://github.com/apache/arrow-rs/issues/8657#issuecomment-3424343012
Seems no. I tried latest main branch and still encountered the error.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Parquet 56: encounter `error: item_reader def levels are None` when reading nested field with row filter [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #8657:
URL: https://github.com/apache/arrow-rs/issues/8657#issuecomment-3534619539
Here is the schema
```
Schema: SchemaDescriptor {
schema: GroupType {
basic_info: BasicTypeInfo {
name: "arrow_schema",
repetition: None,
converted_type: NONE,
logical_type: None,
id: None,
},
fields: [
PrimitiveType {
basic_info: BasicTypeInfo {
name: "id",
repetition: Some(
REQUIRED,
),
converted_type: NONE,
logical_type: None,
id: None,
},
physical_type: INT32,
type_length: -1,
scale: -1,
precision: -1,
},
GroupType {
basic_info: BasicTypeInfo {
name: "vector",
repetition: Some(
OPTIONAL,
),
converted_type: LIST,
logical_type: Some(
List,
),
id: None,
},
fields: [
GroupType {
basic_info: BasicTypeInfo {
name: "list",
repetition: Some(
REPEATED,
),
converted_type: NONE,
logical_type: None,
id: None,
},
fields: [
PrimitiveType {
basic_info: BasicTypeInfo {
name: "item",
repetition: Some(
OPTIONAL,
),
converted_type: NONE,
logical_type: None,
id: None,
},
physical_type: FLOAT,
type_length: -1,
scale: -1,
precision: -1,
},
],
},
],
},
],
},
}
```
here is the mask (no mask == all columns)
```
Projection mask: ProjectionMask {
mask: None,
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Parquet 56: encounter `error: item_reader def levels are None` when reading nested field with row filter [arrow-rs]
Posted by "lewiszlw (via GitHub)" <gi...@apache.org>.
lewiszlw commented on issue #8657:
URL: https://github.com/apache/arrow-rs/issues/8657#issuecomment-3545422748
This workround works on latest main branch, but not works on parquet 56.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Parquet 56: encounter `error: item_reader def levels are None` when reading nested field with row filter [arrow-rs]
Posted by "XiangpengHao (via GitHub)" <gi...@apache.org>.
XiangpengHao commented on issue #8657:
URL: https://github.com/apache/arrow-rs/issues/8657#issuecomment-3481318893
hmmm, I thought nested field will bypass the cache: https://github.com/apache/arrow-rs/blob/8bbcaefcee57aed456055214c4a55056b4588df7/parquet/src/arrow/async_reader/mod.rs#L731-L759
(This is probably the place to fix this)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Parquet 56: encounter `error: item_reader def levels are None` when reading nested field with row filter [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #8657:
URL: https://github.com/apache/arrow-rs/issues/8657#issuecomment-3547905255
I am tracking this down now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Parquet 56: encounter `error: item_reader def levels are None` when reading nested field with row filter [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #8657:
URL: https://github.com/apache/arrow-rs/issues/8657#issuecomment-3548915201
Here is a proposed fix:
- https://github.com/apache/arrow-rs/pull/8866
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Parquet 56: encounter `error: item_reader def levels are None` when reading nested field with row filter [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed issue #8657: Parquet 56: encounter `error: item_reader def levels are None` when reading nested field with row filter
URL: https://github.com/apache/arrow-rs/issues/8657
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Parquet 56: encounter `error: item_reader def levels are None` when reading nested field with row filter [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #8657:
URL: https://github.com/apache/arrow-rs/issues/8657#issuecomment-3547472548
> This workround works on latest main branch, but not works on parquet 56.
I suspect this is due to the fix from @nuno-faria
- https://github.com/apache/arrow-rs/pull/8554
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Parquet 56: encounter `error: item_reader def levels are None` when reading nested field with row filter [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #8657:
URL: https://github.com/apache/arrow-rs/issues/8657#issuecomment-3421585433
I wonder if this is related to this issue (fixed on main/arrow 57):
- https://github.com/apache/arrow-rs/issues/8404
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Parquet 56: encounter `error: item_reader def levels are None` when reading nested field with row filter [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #8657:
URL: https://github.com/apache/arrow-rs/issues/8657#issuecomment-3428790879
@lewiszlw is this a regression? (aka did it work on 55 but started failing on 56?)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Parquet 56: encounter `error: item_reader def levels are None` when reading nested field with row filter [arrow-rs]
Posted by "lewiszlw (via GitHub)" <gi...@apache.org>.
lewiszlw commented on issue #8657:
URL: https://github.com/apache/arrow-rs/issues/8657#issuecomment-3430119305
> @lewiszlw is this a regression? (aka did it work on 55 but started failing on 56?)
Yes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Parquet 56: encounter `error: item_reader def levels are None` when reading nested field with row filter [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #8657:
URL: https://github.com/apache/arrow-rs/issues/8657#issuecomment-3534600807
I have confrmed
```rust
let mut stream = builder
.with_row_filter(RowFilter::new(vec![Box::new(predicate)]))
.with_projection(projection_mask)
.with_max_predicate_cache_size(0) // <-- turning off the predicate cache works around the issue
.build()?;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Parquet 56: encounter `error: item_reader def levels are None` when reading nested field with row filter [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #8657:
URL: https://github.com/apache/arrow-rs/issues/8657#issuecomment-3481266201
I tracked this down to something related to predicate cacheing:
* The test passes at 4a21443d2f
* The test fails at 04f217b670 - #7850
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Parquet 56: encounter `error: item_reader def levels are None` when reading nested field with row filter [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #8657:
URL: https://github.com/apache/arrow-rs/issues/8657#issuecomment-3548003691
I am pretty sure the issue is that [Lists are encoded like this in parquet schema](https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#lists):
```
// List<String> (list non-null, elements nullable)
// required group my_list (LIST) {
// repeated group list {
// optional binary element (STRING);
// }
```
The check for nested types in the parquet cache logic is wrong as it nested types have mulitple children,
but lists only have one child (the "list" group) which in turn has one child (the "element").
I think it is wrong for MAPs as well, but I haven't verified that yet.
The fix is simple, but I am working on figuring out where best to put the tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org