You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "nurpax (via GitHub)" <gi...@apache.org> on 2024/03/26 16:10:56 UTC

[I] How are errors handled when using pq.ParquetWriter to write a bunch of batches [arrow]

nurpax opened a new issue, #40800:
URL: https://github.com/apache/arrow/issues/40800

   ### Describe the usage question you have. Please include as many useful details as  possible.
   
   
   Suppose I have some code like:
   
   ```
   parquet_path = f"dbg/index.parquet"
   with pq.ParquetWriter(parquet_path, schema, compression='snappy', filesystem=s3_fs) as w:
   
   for batch in batches:
       b = pa.RecordBatch.from_pydict(batch, schema=schema)
       w.write_batch(b)
   ```
   
   What if an exception is thrown in the "batches" loop, say the program is sigkilled or the write fails.  Will the result file behind the filesystem be partially written?
   
   I'm asking because on a local file system I guess I'd expect this to be a corrupt file.  In these cases I write to a tmp file and rename on successful writes.
   
   But I'm now planning on writing the parquet to s3.  Will the data be flushed to s3 atomatically, ie., I can trust that either a successful write creates a new object or nothing at all?
   
   ### Component(s)
   
   Python


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] How are errors handled when using pq.ParquetWriter to write a bunch of batches [arrow]

Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on issue #40800:
URL: https://github.com/apache/arrow/issues/40800#issuecomment-2020926672

   :-( Currently, during finish the file abort will called automatically. Maybe you can first handle the exception first 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] How are errors handled when using pq.ParquetWriter to write a bunch of batches [arrow]

Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on issue #40800:
URL: https://github.com/apache/arrow/issues/40800#issuecomment-2021215815

   Currently the behavior is a bit weird. `write` will create a new RowGroup and write to arrow. When doing io, if we teardown during appending new row-group, it will failed here.
   
   When destructoring ParquetWriter, the file would be "close", this would be done syncrhonized.
   
   @jorisvandenbossche Also do we have way to abort the file?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Python] How are errors handled when using pq.ParquetWriter to write a bunch of batches [arrow]

Posted by "nurpax (via GitHub)" <gi...@apache.org>.
nurpax commented on issue #40800:
URL: https://github.com/apache/arrow/issues/40800#issuecomment-2022146767

   > IMHO, this looks hacky to me as it depends on the implementation detail of the underlying file system
   
   It's a little more complicated than that in practice.  The whole processing work is split into thousands of Slurm jobs which individually can take days to weeks to complete.  The jobs may fail due to a number of reasons: canceled by admins, OOM, programming errors, networking, intermittent I/O errors.  I need to keep track of the overall process progress somehow and currently this state is persisted in the filesystem like explained above.  
   
   I can do this safely with the local file system, but as I was planning to save the parquets to s3, I noticed that I may not be able to rely on atomic operation here.  
   
   For reference, here's how I currently save parquets on the filesystem so that the result is either a complete parquet or nothing at all:
   
   ```
       def save_parquet(self, pq_path: str, batches: list[pa.RecordBatch], schema: pa.Schema):
           logging.info(f'Saving database in {pq_path}, {len(batches)} batches')
           try:
               tmp_fd, tmp_name = tempfile.mkstemp(dir=os.path.dirname(pq_path))
               with os.fdopen(tmp_fd, 'wb') as tmpf:
                   with pq.ParquetWriter(tmpf, schema) as writer:
                       for batch in batches:
                           writer.write_batch(batch)
               os.rename(tmp_name, pq_path)
           except:
               logging.exception(f'parquet write to {pq_path} failed')
               os.unlink(tmp_name)
   ```
   
   The idea being that the parquet is written to a temp file and should there be any failures along the way, the result parquet will be deleted.  Otherwise the temp file gets moved to the final parquet file.
   
   As far as I know, an object store like s3 should be able to handle this kind of "atomic write" well.  


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Python] How are errors handled when using pq.ParquetWriter to write a bunch of batches [arrow]

Posted by "thisisnic (via GitHub)" <gi...@apache.org>.
thisisnic commented on issue #40800:
URL: https://github.com/apache/arrow/issues/40800#issuecomment-3547492707

   This issue has been marked as stale because it has had no activity in the past 365 days. Please remove the stale label or comment below, or this issue will be closed in 14 days. If this usage question has evolved into a feature request or docs update, please remove the 'Type: usage' label and add the 'Type: enhancement' label instead.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] How are errors handled when using pq.ParquetWriter to write a bunch of batches [arrow]

Posted by "nurpax (via GitHub)" <gi...@apache.org>.
nurpax commented on issue #40800:
URL: https://github.com/apache/arrow/issues/40800#issuecomment-2020957896

   You mean the below code
   
   ```
   parquet_path = f"dbg/index.parquet"
   with pq.ParquetWriter(parquet_path, schema, compression='snappy', filesystem=s3_fs) as w:
     for batch in batches:
         b = pa.RecordBatch.from_pydict(batch, schema=schema)
         w.write_batch(b)
   ```
   
   would commit even if an exception is thrown?  I'm not sure what's the right way to "close" the file handle without committing any partially written results to s3.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] How are errors handled when using pq.ParquetWriter to write a bunch of batches [arrow]

Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on issue #40800:
URL: https://github.com/apache/arrow/issues/40800#issuecomment-2020996102

   Sigh, yeah, perhaps a explicit "abort" api here would be what you need


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] How are errors handled when using pq.ParquetWriter to write a bunch of batches [arrow]

Posted by "nurpax (via GitHub)" <gi...@apache.org>.
nurpax commented on issue #40800:
URL: https://github.com/apache/arrow/issues/40800#issuecomment-2020896911

   OK.  If the s3 client supports "atomic" writes of an object, then if I have the the full table and write that instead of batches, the write might be atomic?  Or will it still do some sort of a streamed write that may end up being partial?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Python] How are errors handled when using pq.ParquetWriter to write a bunch of batches [arrow]

Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on issue #40800:
URL: https://github.com/apache/arrow/issues/40800#issuecomment-2021987306

   > The reason I'm asking is that I keep track of what parts of my whole dataset have been processed based on the existence of parquet files. If a parquet file is missing, that part of the data would be re-processed.
   
   IMHO, this looks hacky to me as it depends on the implementation detail of the underlying file system, which the IO interface abstraction aims to hide. It would be better to catch any exception on the application side and rerun failed ones explicitly.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Python] How are errors handled when using pq.ParquetWriter to write a bunch of batches [arrow]

Posted by "thisisnic (via GitHub)" <gi...@apache.org>.
thisisnic closed issue #40800: [Python] How are errors handled when using pq.ParquetWriter to write a bunch of batches
URL: https://github.com/apache/arrow/issues/40800


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] How are errors handled when using pq.ParquetWriter to write a bunch of batches [arrow]

Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on issue #40800:
URL: https://github.com/apache/arrow/issues/40800#issuecomment-2020881091

   > What if an exception is thrown in the "batches" loop, say the program is sigkilled or the write fails. Will the result file behind the filesystem be partially written?
   
   It depend on the case, a corrupt file might be written on S3


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] How are errors handled when using pq.ParquetWriter to write a bunch of batches [arrow]

Posted by "nurpax (via GitHub)" <gi...@apache.org>.
nurpax commented on issue #40800:
URL: https://github.com/apache/arrow/issues/40800#issuecomment-2021278575

   I meant using boto3 for uploading the full table to s3. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] How are errors handled when using pq.ParquetWriter to write a bunch of batches [arrow]

Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on issue #40800:
URL: https://github.com/apache/arrow/issues/40800#issuecomment-2020904993

   S3 uses "Part Upload":
   1. Output Stream will change to "part", in S3, a part might be 5MB
   2. During closing the "part", writer might call "commit" to commit the part
   
   I'm not familiar with Python API, but generally, arrow fs also has a "abort", which could abort the file written in parts


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] How are errors handled when using pq.ParquetWriter to write a bunch of batches [arrow]

Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on issue #40800:
URL: https://github.com/apache/arrow/issues/40800#issuecomment-2021239273

   @nurpax If you want to write in RAM, you may face the same problem for S3 part upload 🤔
   https://github.com/apache/arrow/blob/434f87274e8e9adab4f0434ae494f30dc955ca6e/python/pyarrow/parquet/core.py#L1108
   
   maybe an abort interface would be ok here, like:
   
   ```
       def abort(self):
           """
           Close the connection to the Parquet file.
           """
           if self.is_open:
               # ...
           if self.file_handle is not None:
               self.file_handle.abort() # abort S3
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] How are errors handled when using pq.ParquetWriter to write a bunch of batches [arrow]

Posted by "nurpax (via GitHub)" <gi...@apache.org>.
nurpax commented on issue #40800:
URL: https://github.com/apache/arrow/issues/40800#issuecomment-2020910241

   It sounds like at least in theory it might work.  If a multipart upload is never closed (committed), there should be no half-uploaded file.  I'll try with a little test program!  Thanks for the help!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] How are errors handled when using pq.ParquetWriter to write a bunch of batches [arrow]

Posted by "nurpax (via GitHub)" <gi...@apache.org>.
nurpax commented on issue #40800:
URL: https://github.com/apache/arrow/issues/40800#issuecomment-2021198435

   Well, adding a delete adds a potential race condition.  And it's a bit messy when the file just shouldn't show up if the transfer didn't fully complete.
   
   What happens in the case of saving a full table instead of batches, would partial results get committed in that case too?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] How are errors handled when using pq.ParquetWriter to write a bunch of batches [arrow]

Posted by "nurpax (via GitHub)" <gi...@apache.org>.
nurpax commented on issue #40800:
URL: https://github.com/apache/arrow/issues/40800#issuecomment-2021224588

   I guess one way for me to workaround this would be to serialize the parquet into RAM and then write my own s3 code to upload it.  


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org