You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2020/09/08 01:46:26 UTC

[GitHub] [arrow] raduteo opened a new pull request #8130: Parquet file writer snapshot API and proper ColumnChunk.file_path utilization

raduteo opened a new pull request #8130:
URL: https://github.com/apache/arrow/pull/8130


   This is a follow up to the thread:
   https://mail-archives.apache.org/mod_mbox/arrow-dev/202009.mbox/%3cCDD00783-0FFC-4934-AA24-529FB2A44D88@yahoo.com%3e
   
   The specific use case I am targeting is having the ability to partially read a parquet file while it's still being written to.
   This is relevant for any process that is recording events over a long period of times and writing them to parquet (tracing data, logging events or any other live time series)
   The solution relies on the fact that parquet specifications allows column chunk metadata to point explicitly to its location in a file which can theoretically be different from the file containing the metadata (as covered in other threads, this behavior is not fully supported by major parquet implementations).
   My solution is centered around adding a method,
   ```
   void ParquetFileWriter::Snapshot(const std::string& data_path,
                                    std::shared_ptr<::arrow::io::OutputStream>& sink) 
   
   ```
   ,that writes writes the metadata for the RowGroups given so far to the `sink` stream and updates all the ColumnChunk metadata `file_path` to point to `data_path`. This was intended as a minimalist change to `ParquetFileWriter`
   
   On the reading side I implemented full support for ColumnChunk.file_path by introducing `ArrowMultiInputFile` as an alternative to `ArrowInputFile` in the `ParquetFileReader` implementation stack. In the PR implementation one can default to the current behavior by using the `SingleFile` class, have full read support for multi-file parquet in line with the specification by using `MultiReadableFile` implementation (that captures the metafile base directory and uses it as the base directory to the ColumChunk.file_path) or one can provide a separate implementation for a non-posix file system storage. 
   
   For an example see `write_parquet_file_with_snapshot` function in reader-writer.cc that illustrates the snapshotting write while the `read_whole_file` function has been modified to read one of the snapshots (I will rollback that change and provide separate example before the merge)


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] emkornfield commented on pull request #8130: Parquet file writer snapshot API and proper ColumnChunk.file_path utilization

Posted by GitBox <gi...@apache.org>.
emkornfield commented on pull request #8130:
URL: https://github.com/apache/arrow/pull/8130#issuecomment-690000459


   I don't think we should support this unless we can get consensus on dev@parquet mailing list that we want to support this across java and C++ (if java already supports it a pointer would be useful).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] raduteo commented on pull request #8130: Parquet file writer snapshot API and proper ColumnChunk.file_path utilization

Posted by GitBox <gi...@apache.org>.
raduteo commented on pull request #8130:
URL: https://github.com/apache/arrow/pull/8130#issuecomment-690275695


   RowGroup level file name is certainly supported by fastparquet:
   https://github.com/dask/fastparquet/blob/0402257560e20b961a517ee6d770e0995e944163/fastparquet/api.py#L187 <https://github.com/dask/fastparquet/blob/0402257560e20b961a517ee6d770e0995e944163/fastparquet/api.py#L187>
   
   and the java code does read file_path (again with the one-file-per-rowgroup constraint): 
   https://github.com/apache/parquet-mr/blob/65b95fb72be8f5a8a193a6f7bc4560fdcd742fc7/parquet-hadoop/src/main/java/org/apache/parquet/format/converter/ParquetMetadataConverter.java#L1410
   
   but it’s less clear how it is used (certainly ParquetFileReader class seems to stick to a single file)
   
   More than anything the feature is fully in line with the parquet spec (unless I misreading something):
   
   https://github.com/apache/parquet-format/blob/01971a532e20ff8e5eba9d440289bfb753f0cf0b/src/main/thrift/parquet.thrift#L769
   
   Also the code changes are not affecting any of the existing behavior, specifically even if one uses the proposed `Snapshot` method during file writing, the final file is still readable by the java and the fastparquet implementation.
   
   I am happy to open a discussion on the parquet list and push for broader support around this feature, but given that it is spec compliant and backward compatible with the existing code, I hope we can allow this PR to proceeded independently.  
   
   > On Sep 10, 2020, at 1:46 AM, emkornfield <no...@github.com> wrote:
   > 
   > 
   > I don't think we should support this unless we can get consensus on dev@parquet mailing list that we want to support this across java and C++ (if java already supports it a pointer would be useful).
   > 
   > —
   > You are receiving this because you authored the thread.
   > Reply to this email directly, view it on GitHub <https://github.com/apache/arrow/pull/8130#issuecomment-690000459>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ANYOEGKVOLNFHD7HUQBFW23SFBR2XANCNFSM4Q7GLF4A>.
   > 
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] github-actions[bot] commented on pull request #8130: ARROW-11465: [C++] Parquet file writer snapshot API and proper ColumnChunk.file_path utilization

Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #8130:
URL: https://github.com/apache/arrow/pull/8130#issuecomment-771202596


   https://issues.apache.org/jira/browse/ARROW-11465


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] github-actions[bot] commented on pull request #8130: Parquet file writer snapshot API and proper ColumnChunk.file_path utilization

Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #8130:
URL: https://github.com/apache/arrow/pull/8130#issuecomment-688572112


   <!--
     Licensed to the Apache Software Foundation (ASF) under one
     or more contributor license agreements.  See the NOTICE file
     distributed with this work for additional information
     regarding copyright ownership.  The ASF licenses this file
     to you under the Apache License, Version 2.0 (the
     "License"); you may not use this file except in compliance
     with the License.  You may obtain a copy of the License at
   
       http://www.apache.org/licenses/LICENSE-2.0
   
     Unless required by applicable law or agreed to in writing,
     software distributed under the License is distributed on an
     "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
     KIND, either express or implied.  See the License for the
     specific language governing permissions and limitations
     under the License.
   -->
   
   Thanks for opening a pull request!
   
   Could you open an issue for this pull request on JIRA?
   https://issues.apache.org/jira/browse/ARROW
   
   Then could you also rename pull request title in the following format?
   
       ARROW-${JIRA_ID}: [${COMPONENT}] ${SUMMARY}
   
   See also:
   
     * [Other pull requests](https://github.com/apache/arrow/pulls/)
     * [Contribution Guidelines - How to contribute patches](https://arrow.apache.org/docs/developers/contributing.html#how-to-contribute-patches)
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org