You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "asfimport (via GitHub)" <gi...@apache.org> on 2024/06/23 06:32:59 UTC
[I] [C++][Parquet] Implement support for DataPageV2 [arrow]
asfimport opened a new issue, #42256:
URL: https://github.com/apache/arrow/issues/42256
**Reporter**: [Wes McKinney](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=wesm) / @wesm
**Assignee**: Hatem Helal](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=hatem) / @hatemhelal
#### Related issues:
- [[C++] Unable to read parquetjs-created file using low-level parquet-cpp API](https://issues.apache.org/jira/browse/PARQUET-1560) (causes)
- [[Python] Parquet files v2.0 created by spark can't be read by pyarrow](https://issues.apache.org/jira/browse/ARROW-6057) (causes)
#### PRs and other links:
- [GitHub Pull Request #6481](https://github.com/apache/arrow/pull/6481)
<sub>**Note**: *This issue was originally created as [PARQUET-458](https://issues.apache.org/jira/browse/PARQUET-458). Please see the [migration documentation](https://issues.apache.org/jira/browse/PARQUET-2502) for further details.*</sub>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++][Parquet] Implement support for DataPageV2 [arrow]
Posted by "asfimport (via GitHub)" <gi...@apache.org>.
asfimport commented on issue #42256:
URL: https://github.com/apache/arrow/issues/42256#issuecomment-2184204274
[shravani kalikiri](https://issues.apache.org/jira/browse/PARQUET-458?#comment-16895091):
Okay.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++][Parquet] Implement support for DataPageV2 [arrow]
Posted by "asfimport (via GitHub)" <gi...@apache.org>.
asfimport commented on issue #42256:
URL: https://github.com/apache/arrow/issues/42256#issuecomment-2184204282
[Wes McKinney](https://issues.apache.org/jira/browse/PARQUET-458?#comment-16931914) / @wesm:
BTW V2 files are not considered suitable for production by the Parquet community
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++][Parquet] Implement support for DataPageV2 [arrow]
Posted by "asfimport (via GitHub)" <gi...@apache.org>.
asfimport commented on issue #42256:
URL: https://github.com/apache/arrow/issues/42256#issuecomment-2184204271
[shravani kalikiri](https://issues.apache.org/jira/browse/PARQUET-458?#comment-16890972):
Hello @wesm ,
Any update on when this fix will be merged ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++][Parquet] Implement support for DataPageV2 [arrow]
Posted by "asfimport (via GitHub)" <gi...@apache.org>.
asfimport commented on issue #42256:
URL: https://github.com/apache/arrow/issues/42256#issuecomment-2184204263
[Wes McKinney](https://issues.apache.org/jira/browse/PARQUET-458?#comment-16855019) / @wesm:
There's multiple issues here preventing the library from reading the pages yet:
In DataPageV2
- the encoded rep/def levels prefix is not included in the data, it's part of the page header, so this logic is incorrect: https://github.com/apache/arrow/blob/master/cpp/src/parquet/column_reader.cc#L53
- the compressed data size in the page header refers only to the portion of the page after the definition_levels_num_bytes and repetition_levels_num_bytes from the page header
I started working on a patch, I'll see if I can get something up in the next week or so
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++][Parquet] Implement support for DataPageV2 [arrow]
Posted by "asfimport (via GitHub)" <gi...@apache.org>.
asfimport commented on issue #42256:
URL: https://github.com/apache/arrow/issues/42256#issuecomment-2184204281
[Wes McKinney](https://issues.apache.org/jira/browse/PARQUET-458?#comment-16931913) / @wesm:
I have no immediate plans to do this work. Such is the nature of open source software.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++][Parquet] Implement support for DataPageV2 [arrow]
Posted by "asfimport (via GitHub)" <gi...@apache.org>.
asfimport commented on issue #42256:
URL: https://github.com/apache/arrow/issues/42256#issuecomment-2184204284
[Wes McKinney](https://issues.apache.org/jira/browse/PARQUET-458?#comment-17067085) / @wesm:
Issue resolved by pull request 6481
<https://github.com/apache/arrow/pull/6481>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++][Parquet] Implement support for DataPageV2 [arrow]
Posted by "asfimport (via GitHub)" <gi...@apache.org>.
asfimport commented on issue #42256:
URL: https://github.com/apache/arrow/issues/42256#issuecomment-2184204270
[Siva](https://issues.apache.org/jira/browse/PARQUET-458?#comment-16881141):
Thanks Wes Mckinney
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++][Parquet] Implement support for DataPageV2 [arrow]
Posted by "asfimport (via GitHub)" <gi...@apache.org>.
asfimport commented on issue #42256:
URL: https://github.com/apache/arrow/issues/42256#issuecomment-2184204264
[skreddy](https://issues.apache.org/jira/browse/PARQUET-458?#comment-16876703):
Hi Wes Mckinney,
Are you working on fixing this issue ? Any update on when this fix will be available?
Is below reported datapage v2 issue will also be solved with your fix?
https://issues.apache.org/jira/browse/PARQUET-1560
Thanks in advance for your response.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++][Parquet] Implement support for DataPageV2 [arrow]
Posted by "asfimport (via GitHub)" <gi...@apache.org>.
asfimport commented on issue #42256:
URL: https://github.com/apache/arrow/issues/42256#issuecomment-2184204268
[skreddy](https://issues.apache.org/jira/browse/PARQUET-458?#comment-16876823):
Thanks for the response Ivan Sadikov.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++][Parquet] Implement support for DataPageV2 [arrow]
Posted by "asfimport (via GitHub)" <gi...@apache.org>.
asfimport commented on issue #42256:
URL: https://github.com/apache/arrow/issues/42256#issuecomment-2184204269
[Wes McKinney](https://issues.apache.org/jira/browse/PARQUET-458?#comment-16877115) / @wesm:
I do plan to write a C++ patch for this but it will be next week at the earliest. I expect the fix will be a part of the next Arrow release, but you can create builds from master in the meantime once a fix it in
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++][Parquet] Implement support for DataPageV2 [arrow]
Posted by "asfimport (via GitHub)" <gi...@apache.org>.
asfimport commented on issue #42256:
URL: https://github.com/apache/arrow/issues/42256#issuecomment-2184204278
[Wes McKinney](https://issues.apache.org/jira/browse/PARQUET-458?#comment-16902525) / @wesm:
I'd like to fix the underlying issues with the V2 read/write path. I'm not sure if I'll get to it this month but I at least have it on mind. I'm doing some refactoring of parquet/column_writer.cc at the moment but the change to get the format right should not be too large. Making sure it's adequately tested may be as much or more work
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++][Parquet] Implement support for DataPageV2 [arrow]
Posted by "asfimport (via GitHub)" <gi...@apache.org>.
asfimport commented on issue #42256:
URL: https://github.com/apache/arrow/issues/42256#issuecomment-2184204275
[Wes McKinney](https://issues.apache.org/jira/browse/PARQUET-458?#comment-16901491) / @wesm:
@xhochy I think we should disable V2 writes until this can be correctly implemented, what do you think? It's unfortunate that it's been broken for such a long time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++][Parquet] Implement support for DataPageV2 [arrow]
Posted by "asfimport (via GitHub)" <gi...@apache.org>.
asfimport commented on issue #42256:
URL: https://github.com/apache/arrow/issues/42256#issuecomment-2184204283
[Sk](https://issues.apache.org/jira/browse/PARQUET-458?#comment-16932159):
Thanks for the information.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++][Parquet] Implement support for DataPageV2 [arrow]
Posted by "asfimport (via GitHub)" <gi...@apache.org>.
asfimport commented on issue #42256:
URL: https://github.com/apache/arrow/issues/42256#issuecomment-2184204277
[TP Boudreau](https://issues.apache.org/jira/browse/PARQUET-458?#comment-16902497) / @tpboudreau:
Thanks for the heads up.
WRT timestamps, it seems the primary consequence of withdrawing V2 support will be that (Py)Arrow/Parquet users will not be able to create int64 nanosecond timestamps, while non-Arrow parquet-cpp users will still be able to create them.
I suppose we could consider adding a separate Arrow boolean file writer property like "retain_nanosecond_timestamps" that defaulted to "false" (meaning preserve legacy (V1) behavior of coercing nanos to micros) but could be overridden to obtain the current V2 behavior. But this would probably only be worth the effort (and API clutter) if the reintroduction of V2 were a long way off.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++][Parquet] Implement support for DataPageV2 [arrow]
Posted by "asfimport (via GitHub)" <gi...@apache.org>.
asfimport commented on issue #42256:
URL: https://github.com/apache/arrow/issues/42256#issuecomment-2184204265
[Ivan Sadikov](https://issues.apache.org/jira/browse/PARQUET-458?#comment-16876738) / @sadikovi:
Hello,
If it helps, Rust parquet library in Apache Arrow already implements Data page V2 for both writes and reads, so you can use that as a reference when working on C++ code and check correctness in either Rust or C++.
Here are some links for column reader/writer:
https://github.com/apache/arrow/blob/master/rust/parquet/src/column/reader.rs#L342
https://github.com/apache/arrow/blob/master/rust/parquet/src/column/writer.rs#L544
We also added a bunch of Data page v2 encodings added in Rust, and quite a few correctness tests to check encoding/decoding works.
Hopefully, it helps, @wesm.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++][Parquet] Implement support for DataPageV2 [arrow]
Posted by "asfimport (via GitHub)" <gi...@apache.org>.
asfimport commented on issue #42256:
URL: https://github.com/apache/arrow/issues/42256#issuecomment-2184204276
[Wes McKinney](https://issues.apache.org/jira/browse/PARQUET-458?#comment-16901517) / @wesm:
@tpboudreau wanted to make sure you're aware of this issue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++][Parquet] Implement support for DataPageV2 [arrow]
Posted by "asfimport (via GitHub)" <gi...@apache.org>.
asfimport commented on issue #42256:
URL: https://github.com/apache/arrow/issues/42256#issuecomment-2184204272
[Wes McKinney](https://issues.apache.org/jira/browse/PARQUET-458?#comment-16891009) / @wesm:
Seeing as how I haven't written it yet I have no idea
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++][Parquet] Implement support for DataPageV2 [arrow]
Posted by "asfimport (via GitHub)" <gi...@apache.org>.
asfimport commented on issue #42256:
URL: https://github.com/apache/arrow/issues/42256#issuecomment-2184204279
[Sk](https://issues.apache.org/jira/browse/PARQUET-458?#comment-16931137):
@wesm any update on this issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org