You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2021/06/01 13:39:09 UTC
[GitHub] [arrow] yyngwong opened a new pull request #10427: Update ipc.rst
yyngwong opened a new pull request #10427:
URL: https://github.com/apache/arrow/pull/10427
Regarding the necessity to use .close() to close a buffer when not using the .getvalue() function.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #10427: ARROW-12881: [Python] Pyarrow IPC guide is missing sink.close()
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #10427:
URL: https://github.com/apache/arrow/pull/10427#issuecomment-852140873
https://issues.apache.org/jira/browse/ARROW-12881
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on pull request #10427: ARROW-12881: [Python] Pyarrow IPC guide is missing sink.close()
Posted by GitBox <gi...@apache.org>.
pitrou commented on pull request #10427:
URL: https://github.com/apache/arrow/pull/10427#issuecomment-887305825
@amol- What do you think about this PR? Would you like to take it up?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] amol- commented on pull request #10427: ARROW-12881: [Python] Pyarrow IPC guide is missing sink.close()
Posted by GitBox <gi...@apache.org>.
amol- commented on pull request #10427:
URL: https://github.com/apache/arrow/pull/10427#issuecomment-887534590
> @amol- What do you think about this PR? Would you like to take it up?
I wonder if we should foster (in the tutorial/howto) usage of context managers (by getting rid of all examples where `close()` is manually invoked) and reserve the use of explicit `close` for advanced use cases where it's up to the user to know what he/she is doing. At that point you could probably document that the object must be closed in the class documentation itself but you wouldn't have to worry too much in the tutorials because you spread the best practice that will handle closing by itself for the usage docs.
My gut feeling is that this note probably fits more in the class doc itself, but I do also see how striving for perfection might make us delay too much good enough solutions
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on a change in pull request #10427: ARROW-12881: [Python] Pyarrow IPC guide is missing sink.close()
Posted by GitBox <gi...@apache.org>.
lidavidm commented on a change in pull request #10427:
URL: https://github.com/apache/arrow/pull/10427#discussion_r643164275
##########
File path: docs/source/python/ipc.rst
##########
@@ -105,6 +105,11 @@ An important point is that if the input source supports zero-copy reads
(e.g. like a memory map, or ``pyarrow.BufferReader``), then the returned
batches are also zero-copy and do not allocate any new memory on read.
+All stream objects have a ``close`` method. In the above example
+:func:`~pyarrow.BufferOutputStream.getvalue` both closes and returns the buffer.
+If the return object is not needed, call the relevent ``close`` method
+after you are finished using the object to avoid data corruption issues.
Review comment:
This is a little specific to BufferOutputStream (which is used for convenience in this example). I think the relevant point is that closing the writer does not close the stream, so you must close the stream/file yourself to ensure all data is persisted. (And maybe note that in this example, we don't do that as `getvalue` implicitly closes the stream.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on a change in pull request #10427: ARROW-12881: [Python] Pyarrow IPC guide is missing sink.close()
Posted by GitBox <gi...@apache.org>.
pitrou commented on a change in pull request #10427:
URL: https://github.com/apache/arrow/pull/10427#discussion_r643247160
##########
File path: docs/source/python/ipc.rst
##########
@@ -105,6 +105,11 @@ An important point is that if the input source supports zero-copy reads
(e.g. like a memory map, or ``pyarrow.BufferReader``), then the returned
batches are also zero-copy and do not allocate any new memory on read.
+All stream objects have a ``close`` method. In the above example
+:func:`~pyarrow.BufferOutputStream.getvalue` both closes and returns the buffer.
+If the return object is not needed, call the relevent ``close`` method
+after you are finished using the object to avoid data corruption issues.
Review comment:
Also mention that the context protocol should work as well (i.e. the `with` statement).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org