You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2022/06/08 15:30:40 UTC
[GitHub] [arrow] AlvinJ15 opened a new pull request, #13342: [Python] Disable memory mapping by default in pyarrow
AlvinJ15 opened a new pull request, #13342:
URL: https://github.com/apache/arrow/pull/13342
[Python] Disable memory mapping by default in pyarrow
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #13342: ARROW-16382: [Python] Disable memory mapping by default in pyarrow
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #13342:
URL: https://github.com/apache/arrow/pull/13342#issuecomment-1150592513
https://issues.apache.org/jira/browse/ARROW-16382
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] ursabot commented on pull request #13342: ARROW-16382: [Python] Disable memory mapping by default in pyarrow
Posted by GitBox <gi...@apache.org>.
ursabot commented on PR #13342:
URL: https://github.com/apache/arrow/pull/13342#issuecomment-1176957707
Benchmark runs are scheduled for baseline = dc11d4394a51850e0039f9f30940a6a81d3385b7 and contender = 2185f221e1b941f032a546b1f77935665ba4e172. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Scheduled] [ec2-t3-xlarge-us-east-2](https://conbench.ursa.dev/compare/runs/210ff72fdcb1463a919e11e623348ae5...771a882c9b1a4ceca3a9ee6a6141dd3f/)
[Scheduled] [test-mac-arm](https://conbench.ursa.dev/compare/runs/b3c9886c7e824bada777f4fddcb41e82...3763506f894a4daa8d322087b4d9c06b/)
[Scheduled] [ursa-i9-9960x](https://conbench.ursa.dev/compare/runs/c026cdd927314ebc9069a29318f6f79a...eea651d4c0d140d1b722e4e38e61e08f/)
[Scheduled] [ursa-thinkcentre-m75q](https://conbench.ursa.dev/compare/runs/e9f99ef4e3ae4984bb1ad795f9b86d1a...d721d422e44041c4a45dc9254b23ad18/)
Buildkite builds:
[Scheduled] [`2185f221` ec2-t3-xlarge-us-east-2](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ec2-t3-xlarge-us-east-2/builds/1063)
[Scheduled] [`2185f221` test-mac-arm](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-test-mac-arm/builds/1061)
[Scheduled] [`2185f221` ursa-i9-9960x](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-i9-9960x/builds/1046)
[Scheduled] [`2185f221` ursa-thinkcentre-m75q](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-thinkcentre-m75q/builds/1068)
[Finished] [`dc11d439` ec2-t3-xlarge-us-east-2](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ec2-t3-xlarge-us-east-2/builds/919)
[Finished] [`dc11d439` test-mac-arm](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-test-mac-arm/builds/919)
[Finished] [`dc11d439` ursa-i9-9960x](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-i9-9960x/builds/909)
[Finished] [`dc11d439` ursa-thinkcentre-m75q](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-thinkcentre-m75q/builds/921)
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] amol- commented on a diff in pull request #13342: ARROW-16382: [Python] Disable memory mapping by default in pyarrow
Posted by GitBox <gi...@apache.org>.
amol- commented on code in PR #13342:
URL: https://github.com/apache/arrow/pull/13342#discussion_r894378516
##########
python/pyarrow/ipc.pxi:
##########
@@ -399,21 +399,23 @@ cdef class MessageReader(_Weakrefable):
"instead.".format(self.__class__.__name__))
@staticmethod
- def open_stream(source):
+ def open_stream(source, use_memory_map=False):
Review Comment:
I think `user_memory_map` here is misleading for a user.
`source` can be a `PythonFile` or `NativeFile`, in such case the memory map option won't do anything. The option is only effective if the `source` is a string.
Given that the user can pass a `MemoryMappedFile` if they want to explicitly use `memory_map`. I think we should remove the option and explicitly document to use `MemoryMappedFile` as the source if you want to use memory mapping.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] AlvinJ15 commented on pull request #13342: ARROW-16382: [Python] Disable memory mapping by default in pyarrow
Posted by GitBox <gi...@apache.org>.
AlvinJ15 commented on PR #13342:
URL: https://github.com/apache/arrow/pull/13342#issuecomment-1176957627
@ursabot please benchmark
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #13342: ARROW-16382: [Python] Disable memory mapping by default in pyarrow
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #13342:
URL: https://github.com/apache/arrow/pull/13342#issuecomment-1150592521
:warning: Ticket **has no components in JIRA**, make sure you assign one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] amol- commented on a diff in pull request #13342: ARROW-16382: [Python] Disable memory mapping by default in pyarrow
Posted by GitBox <gi...@apache.org>.
amol- commented on code in PR #13342:
URL: https://github.com/apache/arrow/pull/13342#discussion_r894379186
##########
python/pyarrow/ipc.pxi:
##########
@@ -399,21 +399,23 @@ cdef class MessageReader(_Weakrefable):
"instead.".format(self.__class__.__name__))
@staticmethod
- def open_stream(source):
+ def open_stream(source, use_memory_map=False):
Review Comment:
Same applies for the other cases where `_get_input_stream` is used and a `use_memory_map` option was added.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] amol- merged pull request #13342: ARROW-16382: [Python] Disable memory mapping by default in pyarrow
Posted by GitBox <gi...@apache.org>.
amol- merged PR #13342:
URL: https://github.com/apache/arrow/pull/13342
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #13342: [Python] Disable memory mapping by default in pyarrow
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #13342:
URL: https://github.com/apache/arrow/pull/13342#issuecomment-1150073473
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
Thanks for opening a pull request!
If this is not a [minor PR](https://github.com/apache/arrow/blob/master/CONTRIBUTING.md#Minor-Fixes). Could you open an issue for this pull request on JIRA? https://issues.apache.org/jira/browse/ARROW
Opening JIRAs ahead of time contributes to the [Openness](http://theapacheway.com/open/#:~:text=Openness%20allows%20new%20users%20the,must%20happen%20in%20the%20open.) of the Apache Arrow project.
Then could you also rename pull request title in the following format?
ARROW-${JIRA_ID}: [${COMPONENT}] ${SUMMARY}
or
MINOR: [${COMPONENT}] ${SUMMARY}
See also:
* [Other pull requests](https://github.com/apache/arrow/pulls/)
* [Contribution Guidelines - How to contribute patches](https://arrow.apache.org/docs/developers/contributing.html#how-to-contribute-patches)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] amol- commented on a diff in pull request #13342: ARROW-16382: [Python] Disable memory mapping by default in pyarrow
Posted by GitBox <gi...@apache.org>.
amol- commented on code in PR #13342:
URL: https://github.com/apache/arrow/pull/13342#discussion_r896601428
##########
python/pyarrow/ipc.pxi:
##########
@@ -792,10 +793,10 @@ cdef class _RecordBatchStreamReader(RecordBatchReader):
pass
def _open(self, source, IpcReadOptions options=IpcReadOptions(),
- MemoryPool memory_pool=None):
+ MemoryPool memory_pool=None, use_memory_map=False):
Review Comment:
Let's not change the signature of methods, here too one can pass a `MemoryMappedFile`.
```suggestion
MemoryPool memory_pool=None):
```
##########
python/pyarrow/ipc.pxi:
##########
@@ -839,15 +840,15 @@ cdef class _RecordBatchFileReader(_Weakrefable):
def _open(self, source, footer_offset=None,
IpcReadOptions options=IpcReadOptions(),
- MemoryPool memory_pool=None):
+ MemoryPool memory_pool=None, use_memory_map=False):
Review Comment:
```suggestion
MemoryPool memory_pool=None):
```
##########
python/pyarrow/ipc.pxi:
##########
@@ -839,15 +840,15 @@ cdef class _RecordBatchFileReader(_Weakrefable):
def _open(self, source, footer_offset=None,
IpcReadOptions options=IpcReadOptions(),
- MemoryPool memory_pool=None):
+ MemoryPool memory_pool=None, use_memory_map=False):
self.options = options.c_options
self.options.memory_pool = maybe_unbox_memory_pool(memory_pool)
try:
source = as_buffer(source)
except TypeError:
pass
- get_reader(source, True, &self.file)
+ get_reader(source, use_memory_map, &self.file)
Review Comment:
As previous cases it should support `MemoryMappedFile`
```suggestion
get_reader(source, False, &self.file)
```
##########
python/pyarrow/ipc.pxi:
##########
@@ -1075,6 +1076,8 @@ def read_schema(obj, DictionaryMemo dictionary_memo=None):
dictionary_memo : DictionaryMemo, optional
Needed to be able to reconstruct dictionary-encoded fields
with read_record_batch
+ use_memory_map : boolean, default False
+ Use memory mapping when opening file on disk
Review Comment:
```suggestion
```
##########
python/pyarrow/ipc.py:
##########
@@ -44,11 +44,16 @@ class RecordBatchStreamReader(lib._RecordBatchStreamReader):
If None, default values will be used.
memory_pool : MemoryPool, default None
If None, default memory pool is used.
+ use_memory_map : bool, default False
+ If the source is a file path, use a memory map to read file, which can
+ improve performance in some environments.
Review Comment:
Let's add to the `source` parameter documentation that you are expected to use a `MemoryMappedFile` if you want to rely on memory mapping.
```suggestion
```
##########
python/pyarrow/ipc.py:
##########
@@ -44,11 +44,16 @@ class RecordBatchStreamReader(lib._RecordBatchStreamReader):
If None, default values will be used.
memory_pool : MemoryPool, default None
If None, default memory pool is used.
+ use_memory_map : bool, default False
+ If the source is a file path, use a memory map to read file, which can
+ improve performance in some environments.
"""
- def __init__(self, source, *, options=None, memory_pool=None):
+ def __init__(self, source, *, options=None, memory_pool=None,
+ use_memory_map=False):
options = _ensure_default_ipc_read_options(options)
- self._open(source, options=options, memory_pool=memory_pool)
+ self._open(source, options=options, memory_pool=memory_pool,
+ use_memory_map=use_memory_map)
Review Comment:
```suggestion
self._open(source, options=options, memory_pool=memory_pool)
```
##########
python/pyarrow/ipc.py:
##########
@@ -100,13 +105,16 @@ class RecordBatchFileReader(lib._RecordBatchFileReader):
If None, default values will be used.
memory_pool : MemoryPool, default None
If None, default memory pool is used.
+ use_memory_map : boolean, default False
+ Use memory mapping when opening file on disk
"""
def __init__(self, source, footer_offset=None, *, options=None,
- memory_pool=None):
+ memory_pool=None, use_memory_map=False):
options = _ensure_default_ipc_read_options(options)
self._open(source, footer_offset=footer_offset,
- options=options, memory_pool=memory_pool)
+ options=options, memory_pool=memory_pool,
+ use_memory_map=use_memory_map)
Review Comment:
```suggestion
options=options, memory_pool=memory_pool)
```
##########
python/pyarrow/serialization.pxi:
##########
@@ -425,7 +425,7 @@ def serialize_to(object value, sink, SerializationContext context=None):
serialized.write_to(sink)
-def read_serialized(source, base=None):
+def read_serialized(source, base=None, use_memory_map=False):
Review Comment:
```suggestion
def read_serialized(source, base=None):
```
##########
python/pyarrow/serialization.pxi:
##########
@@ -442,18 +442,20 @@ def read_serialized(source, base=None):
base : object
This object will be the base object of all the numpy arrays
contained in the sequence.
+ use_memory_map : boolean, default False
+ Use memory mapping when opening file on disk
Returns
-------
serialized : the serialized data
"""
_deprecate_serialization("read_serialized")
- return _read_serialized(source, base=base)
+ return _read_serialized(source, base=base, use_memory_map=use_memory_map)
Review Comment:
```suggestion
return _read_serialized(source, base=base)
```
##########
python/pyarrow/serialization.pxi:
##########
@@ -485,14 +487,17 @@ def deserialize_from(source, object base, SerializationContext context=None):
contained in the sequence.
context : SerializationContext
Custom serialization and deserialization context.
+ use_memory_map : boolean, default False
+ Use memory mapping when opening file on disk
Returns
-------
object
Python object for the deserialized sequence.
"""
_deprecate_serialization("deserialize_from")
- serialized = _read_serialized(source, base=base)
+ serialized = _read_serialized(
+ source, base=base, use_memory_map=use_memory_map)
Review Comment:
```suggestion
serialized = _read_serialized(source, base=base)
```
##########
python/pyarrow/serialization.pxi:
##########
@@ -550,7 +555,8 @@ def deserialize(obj, SerializationContext context=None):
return _deserialize(obj, context=context)
-def _deserialize(obj, SerializationContext context=None):
+def _deserialize(obj, SerializationContext context=None, use_memory_map=False):
Review Comment:
```suggestion
def _deserialize(obj, SerializationContext context=None):
```
##########
python/pyarrow/ipc.pxi:
##########
@@ -399,21 +399,23 @@ cdef class MessageReader(_Weakrefable):
"instead.".format(self.__class__.__name__))
@staticmethod
- def open_stream(source):
+ def open_stream(source, use_memory_map=False):
Review Comment:
```suggestion
def open_stream(source):
```
##########
python/pyarrow/ipc.pxi:
##########
@@ -562,14 +564,14 @@ cdef class _RecordBatchStreamWriter(_CRecordBatchWriter):
self.options))
-cdef _get_input_stream(object source, shared_ptr[CInputStream]* out):
+cdef _get_input_stream(object source, shared_ptr[CInputStream]* out, use_memory_map=False):
Review Comment:
```suggestion
cdef _get_input_stream(object source, shared_ptr[CInputStream]* out):
```
##########
python/pyarrow/ipc.pxi:
##########
@@ -792,10 +793,10 @@ cdef class _RecordBatchStreamReader(RecordBatchReader):
pass
def _open(self, source, IpcReadOptions options=IpcReadOptions(),
- MemoryPool memory_pool=None):
+ MemoryPool memory_pool=None, use_memory_map=False):
self.options = options.c_options
self.options.memory_pool = maybe_unbox_memory_pool(memory_pool)
- _get_input_stream(source, &self.in_stream)
+ _get_input_stream(source, &self.in_stream, use_memory_map)
Review Comment:
`_get_input_stream` has no `use_memory_map` argument
```suggestion
_get_input_stream(source, &self.in_stream)
```
##########
python/pyarrow/ipc.py:
##########
@@ -44,11 +44,16 @@ class RecordBatchStreamReader(lib._RecordBatchStreamReader):
If None, default values will be used.
memory_pool : MemoryPool, default None
If None, default memory pool is used.
+ use_memory_map : bool, default False
+ If the source is a file path, use a memory map to read file, which can
+ improve performance in some environments.
"""
- def __init__(self, source, *, options=None, memory_pool=None):
+ def __init__(self, source, *, options=None, memory_pool=None,
+ use_memory_map=False):
Review Comment:
```suggestion
def __init__(self, source, *, options=None, memory_pool=None):
```
##########
python/pyarrow/ipc.pxi:
##########
@@ -1089,7 +1092,7 @@ def read_schema(obj, DictionaryMemo dictionary_memo=None):
if isinstance(obj, Message):
raise NotImplementedError(type(obj))
- get_reader(obj, True, &cpp_file)
+ get_reader(obj, use_memory_map, &cpp_file)
Review Comment:
```suggestion
get_reader(obj, False, &cpp_file)
```
##########
python/pyarrow/ipc.pxi:
##########
@@ -1065,7 +1066,7 @@ def read_message(source):
return result
-def read_schema(obj, DictionaryMemo dictionary_memo=None):
+def read_schema(obj, DictionaryMemo dictionary_memo=None, use_memory_map=False):
Review Comment:
```suggestion
def read_schema(obj, DictionaryMemo dictionary_memo=None):
```
This is expected to work on `buffer` or `Message`, none of them can be memory_mapped
##########
python/pyarrow/ipc.py:
##########
@@ -100,13 +105,16 @@ class RecordBatchFileReader(lib._RecordBatchFileReader):
If None, default values will be used.
memory_pool : MemoryPool, default None
If None, default memory pool is used.
+ use_memory_map : boolean, default False
+ Use memory mapping when opening file on disk
"""
def __init__(self, source, footer_offset=None, *, options=None,
- memory_pool=None):
+ memory_pool=None, use_memory_map=False):
Review Comment:
```suggestion
memory_pool=None):
```
##########
python/pyarrow/ipc.pxi:
##########
@@ -562,14 +564,14 @@ cdef class _RecordBatchStreamWriter(_CRecordBatchWriter):
self.options))
-cdef _get_input_stream(object source, shared_ptr[CInputStream]* out):
+cdef _get_input_stream(object source, shared_ptr[CInputStream]* out, use_memory_map=False):
try:
source = as_buffer(source)
except TypeError:
# Non-buffer-like
pass
- get_input_stream(source, True, out)
+ get_input_stream(source, use_memory_map, out)
Review Comment:
```suggestion
get_input_stream(source, False, out)
```
##########
python/pyarrow/ipc.py:
##########
@@ -100,13 +105,16 @@ class RecordBatchFileReader(lib._RecordBatchFileReader):
If None, default values will be used.
memory_pool : MemoryPool, default None
If None, default memory pool is used.
+ use_memory_map : boolean, default False
+ Use memory mapping when opening file on disk
Review Comment:
```suggestion
```
As for previous cases, let's add to `source` documentation that it can be a `MemoryMappedFile`
##########
python/pyarrow/ipc.pxi:
##########
@@ -399,21 +399,23 @@ cdef class MessageReader(_Weakrefable):
"instead.".format(self.__class__.__name__))
@staticmethod
- def open_stream(source):
+ def open_stream(source, use_memory_map=False):
"""
Open stream from source.
Parameters
----------
source
A readable source, like an InputStream
+ use_memory_map : boolean, default False
+ Use memory mapping when opening file on disk
Review Comment:
Let's add to the `source` docstring that you can pass `MemoryMappedFile` if you want to use memory mapping.
##########
python/pyarrow/serialization.pxi:
##########
@@ -442,18 +442,20 @@ def read_serialized(source, base=None):
base : object
This object will be the base object of all the numpy arrays
contained in the sequence.
+ use_memory_map : boolean, default False
+ Use memory mapping when opening file on disk
Returns
-------
serialized : the serialized data
"""
_deprecate_serialization("read_serialized")
- return _read_serialized(source, base=base)
+ return _read_serialized(source, base=base, use_memory_map=use_memory_map)
-def _read_serialized(source, base=None):
+def _read_serialized(source, base=None, use_memory_map=False):
cdef shared_ptr[CRandomAccessFile] stream
- get_reader(source, True, &stream)
+ get_reader(source, use_memory_map, &stream)
Review Comment:
```suggestion
get_reader(source, False, &stream)
```
##########
python/pyarrow/serialization.pxi:
##########
@@ -442,18 +442,20 @@ def read_serialized(source, base=None):
base : object
This object will be the base object of all the numpy arrays
contained in the sequence.
+ use_memory_map : boolean, default False
+ Use memory mapping when opening file on disk
Returns
-------
serialized : the serialized data
"""
_deprecate_serialization("read_serialized")
- return _read_serialized(source, base=base)
+ return _read_serialized(source, base=base, use_memory_map=use_memory_map)
-def _read_serialized(source, base=None):
+def _read_serialized(source, base=None, use_memory_map=False):
Review Comment:
```suggestion
def _read_serialized(source, base=None):
```
##########
python/pyarrow/serialization.pxi:
##########
@@ -442,18 +442,20 @@ def read_serialized(source, base=None):
base : object
This object will be the base object of all the numpy arrays
contained in the sequence.
+ use_memory_map : boolean, default False
+ Use memory mapping when opening file on disk
Review Comment:
```suggestion
```
##########
python/pyarrow/serialization.pxi:
##########
@@ -485,14 +487,17 @@ def deserialize_from(source, object base, SerializationContext context=None):
contained in the sequence.
context : SerializationContext
Custom serialization and deserialization context.
+ use_memory_map : boolean, default False
+ Use memory mapping when opening file on disk
Review Comment:
```suggestion
```
##########
python/pyarrow/serialization.pxi:
##########
@@ -463,7 +465,7 @@ def _read_serialized(source, base=None):
return serialized
-def deserialize_from(source, object base, SerializationContext context=None):
+def deserialize_from(source, object base, SerializationContext context=None, use_memory_map=False):
Review Comment:
```suggestion
def deserialize_from(source, object base, SerializationContext context=None):
```
##########
python/pyarrow/ipc.pxi:
##########
@@ -399,21 +399,23 @@ cdef class MessageReader(_Weakrefable):
"instead.".format(self.__class__.__name__))
@staticmethod
- def open_stream(source):
+ def open_stream(source, use_memory_map=False):
"""
Open stream from source.
Parameters
----------
source
A readable source, like an InputStream
+ use_memory_map : boolean, default False
+ Use memory mapping when opening file on disk
Review Comment:
```suggestion
```
##########
python/pyarrow/ipc.pxi:
##########
@@ -399,21 +399,23 @@ cdef class MessageReader(_Weakrefable):
"instead.".format(self.__class__.__name__))
@staticmethod
- def open_stream(source):
+ def open_stream(source, use_memory_map=False):
"""
Open stream from source.
Parameters
----------
source
A readable source, like an InputStream
+ use_memory_map : boolean, default False
+ Use memory mapping when opening file on disk
"""
cdef:
MessageReader result = MessageReader.__new__(MessageReader)
shared_ptr[CInputStream] in_stream
unique_ptr[CMessageReader] reader
- _get_input_stream(source, &in_stream)
+ _get_input_stream(source, &in_stream, use_memory_map)
Review Comment:
```suggestion
_get_input_stream(source, &in_stream)
```
##########
python/pyarrow/serialization.pxi:
##########
@@ -550,7 +555,8 @@ def deserialize(obj, SerializationContext context=None):
return _deserialize(obj, context=context)
-def _deserialize(obj, SerializationContext context=None):
+def _deserialize(obj, SerializationContext context=None, use_memory_map=False):
source = BufferReader(obj)
- serialized = _read_serialized(source, base=obj)
+ serialized = _read_serialized(
+ source, base=obj, use_memory_map=use_memory_map)
Review Comment:
```suggestion
serialized = _read_serialized(source, base=obj)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org