[I] Provide a way to close a NativeFile without writing the contents [arrow]

["orf (via GitHub)" <gi...@apache.org>]

orf opened a new issue, #40502:
URL: https://github.com/apache/arrow/issues/40502

   ### Describe the enhancement requested
   
   The following code creates an object in S3, even if the exception is raised:
   
   ```python
   from pyarrow.fs import S3FileSystem
   
   fs = S3FileSystem()
   with fs.open_output_stream("bucket/key") as fd:
       fd.write(b"hi")
       raise RuntimeError()
   ```
   
   This is usually what you want (I guess?), but there is no way to change this behaviour. If you have a long-running task that writes to a `NativeFile`, you may not want partially written data to be written to S3 in the case of an error. The workaround is annoying - you need to write to a separate key, then move the file (or delete it):
   
   ```
   fs = S3FileSystem()
   try:
       with fs.open_output_stream("bucket/another_key") as fd:
           fd.write(b"hi")
   except Exception:
       fs.delete("bucket/another_key")
   else:
       fs.move("bucket/another_key", "bucket/actual_key")
   ```
   
   This works, but it's less performant than it should be: data is essentially written to S3 twice, and S3 natively provides a way to prevent this issue: just abort the MultipartUpload (or, don't complete it).
   
   I'd love a way to do something like this:
   
   ```
   fs = S3FileSystem()
   with fs.open_output_stream("bucket/key") as fd:
       fd.write(b"hi")
       fd.abort()
   # The object should now not exist.
   ```
   
   I've tried using `close()` to no avail, and the `background_writes` parameter on the S3FileSystem() seems to have no effect.
   
   ### Component(s)
   
   Python


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Python] Provide a way to close a NativeFile without writing the contents [arrow]

["thisisnic (via GitHub)" <gi...@apache.org>]

thisisnic commented on issue #40502:
URL: https://github.com/apache/arrow/issues/40502#issuecomment-3921308871

   This issue has been closed because there has been no activity for the 14 days since the stale warning. Feel free to reopen if this has been closed in error.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Python] Provide a way to close a NativeFile without writing the contents [arrow]

["thisisnic (via GitHub)" <gi...@apache.org>]

thisisnic commented on issue #40502:
URL: https://github.com/apache/arrow/issues/40502#issuecomment-5869456810

   Thanks for letting us know @adrien-grl - I've reopened it; would you be interested in submitting a PR?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Python] Provide a way to close a NativeFile without writing the contents [arrow]

["github-actions[bot] (via GitHub)" <gi...@apache.org>]

github-actions[bot] commented on issue #40502:
URL: https://github.com/apache/arrow/issues/40502#issuecomment-3547078797

   This issue has been marked as stale because it has had no activity in the past 365 days. Please remove the stale label or comment below, or this issue will be closed in 14 days. If this improvement is still desired but has no current owner, please add the 'Status: needs champion' label.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Python] Provide a way to close a NativeFile without writing the contents [arrow]

["thisisnic (via GitHub)" <gi...@apache.org>]

thisisnic closed issue #40502: [Python] Provide a way to close a NativeFile without writing the contents
URL: https://github.com/apache/arrow/issues/40502


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Python] Provide a way to close a NativeFile without writing the contents [arrow]

["adrien-grl (via GitHub)" <gi...@apache.org>]

adrien-grl commented on issue #40502:
URL: https://github.com/apache/arrow/issues/40502#issuecomment-5868414746

   Hello, this is a feature we would have loved! It would be nice if it could be opened again.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org