You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "b4l (via GitHub)" <gi...@apache.org> on 2023/11/08 14:20:13 UTC
[I] Parquet error: Invalid offset in sparse column chunk data [arrow-datafusion]
b4l opened a new issue, #8092:
URL: https://github.com/apache/arrow-datafusion/issues/8092
### Describe the bug
Querying on a registered parquet table sometimes throws an error. The query is doing a range query over multiple fields. The same query with different literals, in which only one range impacts the output as the others are unbound works fine. Also, this happens for SQL and DataFrame alike.
### To Reproduce
_No response_
### Expected behavior
Query returns result.
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Parquet error: Invalid offset in sparse column chunk data [arrow-datafusion]
Posted by "liukun4515 (via GitHub)" <gi...@apache.org>.
liukun4515 commented on issue #8092:
URL: https://github.com/apache/arrow-datafusion/issues/8092#issuecomment-1857178197
We should add some comment to this issue or the related pr.
If user don't want to update the arrow-rs or the datafusion version to resolve this issue, they can disable the page index:
set the `enable_page_index` to `false`. to skip this bug.
cc @alamb @tustvold
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Parquet error: Invalid offset in sparse column chunk data [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #8092:
URL: https://github.com/apache/arrow-datafusion/issues/8092#issuecomment-1808779367
Please let us know if that doesn't work
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Parquet error: Invalid offset in sparse column chunk data [arrow-datafusion]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on issue #8092:
URL: https://github.com/apache/arrow-datafusion/issues/8092#issuecomment-1802040345
This should have been fixed by https://github.com/apache/arrow-rs/issues/5036
Perhaps you could try out #8029
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Parquet error: Invalid offset in sparse column chunk data [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed issue #8092: Parquet error: Invalid offset in sparse column chunk data
URL: https://github.com/apache/arrow-datafusion/issues/8092
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Parquet error: Invalid offset in sparse column chunk data [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #8092:
URL: https://github.com/apache/arrow-datafusion/issues/8092#issuecomment-1857762595
> If user don't want to update the arrow-rs or the datafusion version to resolve this issue, they can disable the page index:
> set the `enable_page_index` to `false`. to skip this bug.
That is a good point
The workaround suggested by @liukun4515 can be implemented like this:
### SQL
```sql
❯ set datafusion.execution.parquet.enable_page_index = false;
0 rows in set. Query took 0.000 seconds.
```
### Programatically
By setting [`ParquetOptions::enable_page_index`](https://docs.rs/datafusion/latest/datafusion/common/config/struct.ParquetOptions.html#structfield.enable_page_index) to false
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Parquet error: Invalid offset in sparse column chunk data [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #8092:
URL: https://github.com/apache/arrow-datafusion/issues/8092#issuecomment-1808779000
FWIW we have released arrow 48.0.1 (see https://github.com/apache/arrow-rs/issues/5050) so a `cargo update` in your project should get the fix @b4l
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Parquet error: Invalid offset in sparse column chunk data [arrow-datafusion]
Posted by "b4l (via GitHub)" <gi...@apache.org>.
b4l commented on issue #8092:
URL: https://github.com/apache/arrow-datafusion/issues/8092#issuecomment-1802200658
@tustvold thanks! It is indeed working as expected with #8029. Looking forward to the release.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org