You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2022/04/13 10:37:45 UTC
[GitHub] [arrow-rs] tustvold opened a new pull request, #1555: fix infinite loop in not fully packed bit-packed runs
tustvold opened a new pull request, #1555:
URL: https://github.com/apache/arrow-rs/pull/1555
# Which issue does this PR close?
Closes #1548
Cherry-picked from #1460, will add test and then mark ready for review
# Rationale for this change
See ticket
# What changes are included in this PR?
See ticket
# Are there any user-facing changes?
No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] alamb commented on pull request #1555: fix infinite loop in not fully packed bit-packed runs
Posted by GitBox <gi...@apache.org>.
alamb commented on PR #1555:
URL: https://github.com/apache/arrow-rs/pull/1555#issuecomment-1100120390
I plan to merge this after all (non miri) tests pass
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] alamb commented on a diff in pull request #1555: fix infinite loop in not fully packed bit-packed runs
Posted by GitBox <gi...@apache.org>.
alamb commented on code in PR #1555:
URL: https://github.com/apache/arrow-rs/pull/1555#discussion_r851262306
##########
parquet/src/encodings/rle.rs:
##########
@@ -743,6 +753,42 @@ mod tests {
}
}
+ #[test]
+ fn test_truncated_rle() {
Review Comment:
FWIW this test times out without the code in this PR 👍 which is a good sign to me that it covers the issue
```
test encodings::rle::tests::test_truncated_rle has been running for over 60 seconds
```
(I also tried removing each of the two cases -- dict and non dict and the test hung in both)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] alamb merged pull request #1555: fix infinite loop in not fully packed bit-packed runs
Posted by GitBox <gi...@apache.org>.
alamb merged PR #1555:
URL: https://github.com/apache/arrow-rs/pull/1555
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] tustvold commented on pull request #1555: fix infinite loop in not fully packed bit-packed runs
Posted by GitBox <gi...@apache.org>.
tustvold commented on PR #1555:
URL: https://github.com/apache/arrow-rs/pull/1555#issuecomment-1100107482
> I found one in the DeltaBitPackedEncoder that maybe has a similar problem:
In this case the specification is very clear that the miniblock can't be truncated
> If there are not enough values to fill the last miniblock, we pad the miniblock so that its length is always the number of values in a full miniblock multiplied by the bit width. The values of the padding bits should be zero, but readers must accept paddings consisting of arbitrary bits as well.
And we return an error a line below if it is, so I think we should be ok.
Good shout to check though :+1:
The other cases don't appear to have loops, and GenericColumnReader which drives them will bail out if a page is truncated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] alamb commented on pull request #1555: fix infinite loop in not fully packed bit-packed runs
Posted by GitBox <gi...@apache.org>.
alamb commented on PR #1555:
URL: https://github.com/apache/arrow-rs/pull/1555#issuecomment-1100096872
fyi @anliakho2
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org