You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@parquet.apache.org by "jp0317 (via GitHub)" <gi...@apache.org> on 2024/08/13 23:35:07 UTC
[PR] adding some bad parquet files [parquet-testing]
jp0317 opened a new pull request, #58:
URL: https://github.com/apache/parquet-testing/pull/58
Hi, these two bad files can help reproduce the two examples mentioned in https://github.com/apache/arrow-rs/issues/6228.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] adding some bad parquet files [parquet-testing]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #58:
URL: https://github.com/apache/parquet-testing/pull/58#issuecomment-2292275471
> So maybe arrow-rs can also check this? ( also cc @alamb )
I agree we should add tests to parquet-rs to test these bad files. I'll see what i can do
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] adding some bad parquet files [parquet-testing]
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU merged PR #58:
URL: https://github.com/apache/parquet-testing/pull/58
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] adding some bad parquet files [parquet-testing]
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on PR #58:
URL: https://github.com/apache/parquet-testing/pull/58#issuecomment-2290987998
@pitrou Would you mind check the naming of the 2 new bad file here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] adding some bad parquet files [parquet-testing]
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on PR #58:
URL: https://github.com/apache/parquet-testing/pull/58#issuecomment-2290985490
@jp0317 I've check the bad-dict file, it throw here in C++ code:
```
if (col_start < 0 || col_length < 0) {
throw ParquetException("Invalid column metadata (corrupt file?)");
}
if (AddWithOverflow(col_start, col_length, &col_end) || col_end > source_size) {
throw ParquetException("Invalid column metadata (corrupt file?)");
}
```
So maybe arrow-rs can also check this? ( also cc @alamb )
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] adding some bad parquet files [parquet-testing]
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on PR #58:
URL: https://github.com/apache/parquet-testing/pull/58#issuecomment-2288250687
Just wonder how these file is generated? 🤔 Seems we have some "little" file here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] adding some bad parquet files [parquet-testing]
Posted by "jp0317 (via GitHub)" <gi...@apache.org>.
jp0317 commented on PR #58:
URL: https://github.com/apache/parquet-testing/pull/58#issuecomment-2292729477
> @jp0317 I've check the bad-dict file, it throw here in C++ code:
>
> ```
> if (col_start < 0 || col_length < 0) {
> throw ParquetException("Invalid column metadata (corrupt file?)");
> }
>
> if (AddWithOverflow(col_start, col_length, &col_end) || col_end > source_size) {
> throw ParquetException("Invalid column metadata (corrupt file?)");
> }
> ```
>
> So maybe arrow-rs can also check this? ( also cc @alamb )
yes, iiuc we can first improve type conversions as mentioned https://github.com/apache/arrow-rs/issues/6228. For this particular case the lenth is negative but currently our rust codes just convert them as unsigned integers directly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] adding some bad parquet files [parquet-testing]
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on PR #58:
URL: https://github.com/apache/parquet-testing/pull/58#issuecomment-2292817901
I've an issue here: https://github.com/apache/arrow-rs/issues/6255 , glad to learn some rust code lol
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] adding some bad parquet files [parquet-testing]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #58:
URL: https://github.com/apache/parquet-testing/pull/58#issuecomment-2292365710
> So maybe arrow-rs can also check this? ( also cc @alamb )
I made a PR to check these files with Rust: https://github.com/apache/arrow-rs/pull/6262
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org