You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "senyosimpson (via GitHub)" <gi...@apache.org> on 2025/01/24 17:26:31 UTC
[I] Querying Parquet file specifically with a predicate returns invalid data error but works in other situations [datafusion]
senyosimpson opened a new issue, #14281:
URL: https://github.com/apache/datafusion/issues/14281
### Describe the bug
When making a query _with a predicate_ against Parquet files generated with [parquet-go](https://github.com/parquet-go/parquet-go) , DataFusion errors saying the data is invalid. However, without a predicate, it works fine.
When using the CLI, I get the error:
```
» datafusion-cli --command "select * from 'go-parquet-writer/go-testfile.parquet' where age > 10"
DataFusion CLI v44.0.0
Error: External error: Parquet error: External: bad data
```
In my application, it is more descriptive, showing:
```
ParquetError(External(ProtocolError { kind: InvalidData, message: "cannot convert 2 into TType" }))
```
However, it appears that the file is intact. The metadata is successfully read and interpreted
```
» datafusion-cli --command "describe 'go-parquet-writer/go-testfile.parquet'"
DataFusion CLI v44.0.0
+---------------+-------------------------------------+-------------+
| column_name | data_type | is_nullable |
+---------------+-------------------------------------+-------------+
| city | Utf8View | NO |
| country | Utf8View | NO |
| age | UInt8 | NO |
| scale | Int16 | NO |
| status | UInt32 | NO |
| time_captured | Timestamp(Millisecond, Some("UTC")) | NO |
| checked | Boolean | NO |
+---------------+-------------------------------------+-------------+
7 row(s) fetched.
Elapsed 0.001 seconds.
```
When I run without a predicate, I get back the data
```
» datafusion-cli --command "select * from 'go-parquet-writer/go-testfile.parquet'"
DataFusion CLI v44.0.0
+--------+---------+-----+-------+--------+--------------------------+---------+
| city | country | age | scale | status | time_captured | checked |
+--------+---------+-----+-------+--------+--------------------------+---------+
| Madrid | Spain | 10 | -1 | 12 | 2025-01-24T16:34:00.715Z | false |
| Athens | Greece | 32 | 1 | 20 | 2025-01-24T17:34:00.715Z | true |
+--------+---------+-----+-------+--------+--------------------------+---------+
2 row(s) fetched.
Elapsed 0.002 seconds.
```
It even works if I use `ORDER BY` and `GROUP BY`
```
» datafusion-cli --command "select * from 'go-parquet-writer/go-testfile.parquet' ORDER BY age DESC"
DataFusion CLI v44.0.0
+--------+---------+-----+-------+--------+--------------------------+---------+
| city | country | age | scale | status | time_captured | checked |
+--------+---------+-----+-------+--------+--------------------------+---------+
| Athens | Greece | 32 | 1 | 20 | 2025-01-24T17:34:00.715Z | true |
| Madrid | Spain | 10 | -1 | 12 | 2025-01-24T16:34:00.715Z | false |
+--------+---------+-----+-------+--------+--------------------------+---------+
2 row(s) fetched.
Elapsed 0.010 seconds.
» datafusion-cli --command "select city, SUM(age) AS age from 'go-parquet-writer/go-testfile.parquet' GROUP BY city"
DataFusion CLI v44.0.0
+--------+-----+
| city | age |
+--------+-----+
| Athens | 32 |
| Madrid | 10 |
+--------+-----+
2 row(s) fetched.
Elapsed 0.004 seconds.
```
Additionally, this works when I use `PyArrow` and `Pandas` to load the Parquet file and filter it.
### To Reproduce
The issue can be reproduced by creating a Parquet file with the `parquet-go` library and attempting to query it with a predicate in the query. To simplify, I created a [public repo](https://github.com/senyosimpson/fusion-repro) that has code to generate the file and similar examples in the README as shown in this report. A test file can be found in `go-parquet-writer/go-testfile.parquet`, generated by the Go program in that directory.
I've also gone through the effort of trying to achieve the same using PyArrow and Pandas (which you'll see in the repo under `pyarrow-ex`) to verify the Parquet file is not corrupted in some way. This works as expected.
### Expected behavior
The parquet file's created by `parquet-go` can successfully be queried when the query contains a predicate.
### Additional context
From everything I've gathered, this error is likely coming from this [conversion function](https://github.com/apache/thrift/blob/7734c393ed0f0632c658c05e33a4d6592cf2912c/lib/rs/src/protocol/compact.rs#L660-L679). However, it only skips checking `0x02` when a [collection is being parsed](https://github.com/apache/thrift/blob/7734c393ed0f0632c658c05e33a4d6592cf2912c/lib/rs/src/protocol/compact.rs#L653-L658). Weirdly, I don't have any list/map/set in my schema. I assume this means this `0x02` is being used to encode something else but it is beyond my knowledge.
I went spelunking in `parquet-go` codebase. The Thrift protocol implementation is split amongst [the compact protocol](https://github.com/parquet-go/parquet-go/blob/main/encoding/thrift/compact.go), [the Thrift type definitions](https://github.com/parquet-go/parquet-go/blob/main/encoding/thrift/thrift.go) and [the encoding logic](https://github.com/parquet-go/parquet-go/blob/main/encoding/thrift/encode.go)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Querying Parquet file specifically with a predicate returns invalid data error but works in other situations [datafusion]
Posted by "jhorstmann (via GitHub)" <gi...@apache.org>.
jhorstmann commented on issue #14281:
URL: https://github.com/apache/datafusion/issues/14281#issuecomment-2628483026
I think that error message is now related to the values in the list, previous one was about the type of the list. I guess we need to be a bit more lenient for the values too. I'll update the PR.
For reference, the documentation was updated in
https://github.com/apache/thrift/commit/2c29c5665bc442e703480bb0ee60fe925ffe02e8 but only spells out the compatibility issue for element types.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Querying Parquet file specifically with a predicate returns invalid data error but works in other situations [datafusion]
Posted by "kosiew (via GitHub)" <gi...@apache.org>.
kosiew commented on issue #14281:
URL: https://github.com/apache/datafusion/issues/14281#issuecomment-2615424170
I used duckdb to copy go-testfile.parquet to go-testfile3.parquet
D create table data1 as select * from 'go-testfile.parquet';
D copy data1 to 'go-testfile3.parquet' (format 'parquet');
```
DataFusion CLI v44.0.0
> select * from 'tests/data/go-testfile3.parquet' where age > 10;
+--------+---------+-----+-------+--------+--------------------------+---------+
| city | country | age | scale | status | time_captured | checked |
+--------+---------+-----+-------+--------+--------------------------+---------+
| Athens | Greece | 32 | 1 | 20 | 2025-01-24T17:34:00.715Z | true |
+--------+---------+-----+-------+--------+--------------------------+---------+
1 row(s) fetched.
Elapsed 0.014 seconds.
> select * from 'tests/data/go-testfile.parquet' where age > 10;
External error: Parquet error: External: bad data
```
From tests, it looks like the issue is with go-parquet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Querying Parquet file specifically with a predicate returns invalid data error but works in other situations [datafusion]
Posted by "senyosimpson (via GitHub)" <gi...@apache.org>.
senyosimpson commented on issue #14281:
URL: https://github.com/apache/datafusion/issues/14281#issuecomment-2628243005
@alamb i gave it a try and ended up with a different error but i think that hints towards the metadata being wrong. This is the error
```
called `Result::unwrap()` on an `Err` value: External(ProtocolError { kind: InvalidData, message: "cannot convert 0 into bool" })
```
which is coming from here: https://github.com/apache/arrow-rs/blob/3bf29a2c7474e59722d885cd11fafd0dca13a28e/parquet/src/thrift.rs#L191-L206
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Querying Parquet file specifically with a predicate returns invalid data error but works in other situations [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed issue #14281: Querying Parquet file specifically with a predicate returns invalid data error but works in other situations
URL: https://github.com/apache/datafusion/issues/14281
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Querying Parquet file specifically with a predicate returns invalid data error but works in other situations [datafusion]
Posted by "senyosimpson (via GitHub)" <gi...@apache.org>.
senyosimpson commented on issue #14281:
URL: https://github.com/apache/datafusion/issues/14281#issuecomment-2620214852
Interestingly enough, I can run a query on this just fine using duckdb
```
» duckdb
v1.1.3 19864453f7
Enter ".help" for usage hints.
Connected to a transient in-memory database.
Use ".open FILENAME" to reopen on a persistent database.
D select * from 'go-testfile.parquet' where age > 10;
┌─────────┬─────────┬───────┬───────┬────────┬────────────────────────────┬─────────┐
│ city │ country │ age │ scale │ status │ time_captured │ checked │
│ varchar │ varchar │ uint8 │ int16 │ uint32 │ timestamp with time zone │ boolean │
├─────────┼─────────┼───────┼───────┼────────┼────────────────────────────┼─────────┤
│ Athens │ Greece │ 32 │ 1 │ 20 │ 2025-01-25 13:13:48.332+02 │ true │
└─────────┴─────────┴───────┴───────┴────────┴────────────────────────────┴─────────┘
```
And it has the format version you'd expect
```
D select file_name, created_by, format_version from parquet_file_metadata('go-testfile.parquet');
┌─────────────────────┬─────────────────────────────────────────────────────────┬────────────────┐
│ file_name │ created_by │ format_version │
│ varchar │ varchar │ int64 │
├─────────────────────┼─────────────────────────────────────────────────────────┼────────────────┤
│ go-testfile.parquet │ github.com/parquet-go/parquet-go version 0.24.0(build ) │ 2 │
└─────────────────────┴─────────────────────────────────────────────────────────┴────────────────┘
```
However, if I use PyArrow and write out a file with the version format 2.6, datafusion works fine. So I'm still not sure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Querying Parquet file specifically with a predicate returns invalid data error but works in other situations [datafusion]
Posted by "senyosimpson (via GitHub)" <gi...@apache.org>.
senyosimpson commented on issue #14281:
URL: https://github.com/apache/datafusion/issues/14281#issuecomment-2613786689
> Have you tried using the `parquet_metadata` function on the file to see if it returns anything? For example:
>
> datafusion-cli --command "select * from parquet_metadata('go-parquet-writer/go-testfile.parquet')"
That seems to works fine
```
» datafusion-cli --command "select path_in_schema, stats_min, stats_max from parquet_metadata('go-parquet-writer/go-testfile.parquet')"
DataFusion CLI v44.0.0
+-----------------+---------------+---------------+
| path_in_schema | stats_min | stats_max |
+-----------------+---------------+---------------+
| "city" | Athens | Madrid |
| "country" | Greece | Spain |
| "age" | 10 | 32 |
| "scale" | -1 | 1 |
| "status" | 12 | 20 |
| "time_captured" | 1737781788268 | 1737785388268 |
| "checked" | false | true |
+-----------------+---------------+---------------+
7 row(s) fetched.
Elapsed 0.003 seconds.
```
> Hey, I'm the one who mentioned this in discord 😄 what I meant is that we disabled `datafusion.execution.parquet.enable_page_index` at query time, not that we skipped generating the page index when generating the parquet file.
Ah right, that makes sense. Thanks for the help on getting that to work, that's super helpful!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Querying Parquet file specifically with a predicate returns invalid data error but works in other situations [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #14281:
URL: https://github.com/apache/datafusion/issues/14281#issuecomment-2627834027
@jhorstmann made a fix:
- https://github.com/apache/datafusion/issues/14281
@senyosimpson any chance you can verify that the change in https://github.com/apache/datafusion/issues/14281 fixes your issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Querying Parquet file specifically with a predicate returns invalid data error but works in other situations [datafusion]
Posted by "senyosimpson (via GitHub)" <gi...@apache.org>.
senyosimpson commented on issue #14281:
URL: https://github.com/apache/datafusion/issues/14281#issuecomment-2613908545
Confirmed that the following works now.
```rust
let mut parquet_options = TableParquetOptions::new();
parquet_options
.set("enable_page_index", "false")
.expect("could not set enable_page_index config option");
let exec = ParquetExec::builder(scan_config)
.with_table_parquet_options(parquet_options)
.with_predicate(predicate)
.build_arc();
```
Weirdly enough, setting the variable on the `SessionContext` doesn't work? Is that the correct behaviour? I'm sure I had it set right via
```rust
let ctx = SessionContext::new_with_config(
SessionConfig::new().set_bool("datafusion.execution.parquet.enable_page_index", false)
)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Querying Parquet file specifically with a predicate returns invalid data error but works in other situations [datafusion]
Posted by "senyosimpson (via GitHub)" <gi...@apache.org>.
senyosimpson commented on issue #14281:
URL: https://github.com/apache/datafusion/issues/14281#issuecomment-2628828260
@jhorstmann tested again and works now! I'm not sure how to test if the querying works seamlessly but reading the metadata works now so i'm assuming everything upstream will work fine!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Querying Parquet file specifically with a predicate returns invalid data error but works in other situations [datafusion]
Posted by "kosiew (via GitHub)" <gi...@apache.org>.
kosiew commented on issue #14281:
URL: https://github.com/apache/datafusion/issues/14281#issuecomment-2626265549
Adding findings:
The error
https://github.com/apache/arrow-rs/blob/3bf29a2c7474e59722d885cd11fafd0dca13a28e/parquet/src/thrift.rs#L284-L298
is raised when trying to load the metadata
https://github.com/apache/datafusion/blob/53728b30aa95be7865a0ad35e70ec574312a69bc/datafusion/core/src/datasource/physical_plan/parquet/opener.rs#L126-L130
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Querying Parquet file specifically with a predicate returns invalid data error but works in other situations [datafusion]
Posted by "senyosimpson (via GitHub)" <gi...@apache.org>.
senyosimpson commented on issue #14281:
URL: https://github.com/apache/datafusion/issues/14281#issuecomment-2613567930
I've heard from another user that they managed to work around this by switching off the page index when generating the files by `parquet-go`. However, when I tried this, I still ran into this problem
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Querying Parquet file specifically with a predicate returns invalid data error but works in other situations [datafusion]
Posted by "jhorstmann (via GitHub)" <gi...@apache.org>.
jhorstmann commented on issue #14281:
URL: https://github.com/apache/datafusion/issues/14281#issuecomment-2628535178
@senyosimpson @alamb Updated the PR, would be great to get another test run with that file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Querying Parquet file specifically with a predicate returns invalid data error but works in other situations [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #14281:
URL: https://github.com/apache/datafusion/issues/14281#issuecomment-2629375060
Maybe what we can do is once the arrow-rs fix is in, we can add a test in datafusion with this particular test file / query to ensure there are no regressions
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Querying Parquet file specifically with a predicate returns invalid data error but works in other situations [datafusion]
Posted by "matthewmturner (via GitHub)" <gi...@apache.org>.
matthewmturner commented on issue #14281:
URL: https://github.com/apache/datafusion/issues/14281#issuecomment-2613711782
Have you tried using the `parquet_metadata` function on the file? For example:
```bash
datafusion-cli --command "select * from parquet_metadata('go-parquet-writer/go-testfile.parquet')"
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Querying Parquet file specifically with a predicate returns invalid data error but works in other situations [datafusion]
Posted by "suremarc (via GitHub)" <gi...@apache.org>.
suremarc commented on issue #14281:
URL: https://github.com/apache/datafusion/issues/14281#issuecomment-2613757948
> I've heard from another user that they managed to work around this by switching off the page index when generating the files by `parquet-go`. However, when I tried this, I still ran into this problem
Hey, I'm the one who mentioned this in discord 😄 what I meant is that we disabled `datafusion.execution.parquet.enable_page_index` at query time, not that we skipped generating the page index when generating the parquet file.
In your [repro repository](https://github.com/senyosimpson/fusion-repro) I was able to confirm disabling the page index makes it work:
```sql
❯ datafusion-cli
DataFusion CLI v43.0.0
> select * from 'go-parquet-writer/go-testfile.parquet' where age > 10;
External error: Parquet error: External: bad data
> SET datafusion.execution.parquet.enable_page_index = false;
0 row(s) fetched.
Elapsed 0.001 seconds.
> select * from 'go-parquet-writer/go-testfile.parquet' where age > 10;
+--------+---------+-----+-------+--------+--------------------------+---------+
| city | country | age | scale | status | time_captured | checked |
+--------+---------+-----+-------+--------+--------------------------+---------+
| Athens | Greece | 32 | 1 | 20 | 2025-01-24T17:34:00.715Z | true |
+--------+---------+-----+-------+--------+--------------------------+---------+
1 row(s) fetched.
Elapsed 0.021 seconds.
```
Last time I looked at this issue I had a feeling that this was an issue with `parquet-go`'s Thrift implementation but I wasn't able to find evidence, or tbh I also just don't remember since it's been quite some time... It's possible that pyarrow and pandas work if they aren't utilizing the page index for predicate pushdown.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Querying Parquet file specifically with a predicate returns invalid data error but works in other situations [datafusion]
Posted by "senyosimpson (via GitHub)" <gi...@apache.org>.
senyosimpson commented on issue #14281:
URL: https://github.com/apache/datafusion/issues/14281#issuecomment-2626439477
> Since `0x02` is not supported in arrow-rs
What's interesting here is that function seems to be a helper function. `0x02` is handled here when reading a field for example: https://github.com/apache/arrow-rs/blob/3bf29a2c7474e59722d885cd11fafd0dca13a28e/parquet/src/thrift.rs#L131-L147.
But when there's a collection, it is skipped: https://github.com/apache/arrow-rs/blob/3bf29a2c7474e59722d885cd11fafd0dca13a28e/parquet/src/thrift.rs#L275-L280
But there's no collections in the example schemas which points to another instance that function is being used.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Querying Parquet file specifically with a predicate returns invalid data error but works in other situations [datafusion]
Posted by "jhorstmann (via GitHub)" <gi...@apache.org>.
jhorstmann commented on issue #14281:
URL: https://github.com/apache/datafusion/issues/14281#issuecomment-2627342594
Having looked into thrift some time ago, this reminds me of this [inconsistency about boolean encodings](https://github.com/jhorstmann/compact-thrift/issues/7), where boolean values would be encoded as `2` contrary to the documentation.
But that was about values, here it seems the go implementation writes the wrong tag in the header of a list field. With my limited go knowledge, I think the error might be in https://github.com/parquet-go/parquet-go/blob/e0b932aa3784539421f0afb46d81cbc4395f4f74/encoding/thrift/thrift.go#L64
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Querying Parquet file specifically with a predicate returns invalid data error but works in other situations [datafusion]
Posted by "senyosimpson (via GitHub)" <gi...@apache.org>.
senyosimpson commented on issue #14281:
URL: https://github.com/apache/datafusion/issues/14281#issuecomment-2618599321
Ah, interesting, thanks for that!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org