You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@arrow.apache.org by Сергей Красовский <kr...@gmail.com> on 2021/01/26 07:45:14 UTC
[Python] HDFS write fails when size of file is higher than 6gb
Hello Arrow team,
I have an issue with writing files with size > 6143mb to HDFS. Exception is:
Traceback (most recent call last):
> File "exp.py", line 22, in <module>
> output_stream.write(open(source, "rb").read())
> File "pyarrow/io.pxi", line 283, in pyarrow.lib.NativeFile.write
> File "pyarrow/error.pxi", line 99, in pyarrow.lib.check_status
> OSError: HDFS Write failed, errno: 22 (Invalid argument)
>
The code below works for files with size <= 6143mb.
Hadoop version: 3.1.1.3.1.4.0-315
Python version: 3.6.10
Pyarrow version: 2.0.0
System: Ubuntu 16.04.7 LTS
I try to understand what happens under the hood
of pyarrow.lib.NativeFile.write. Is there any limitation from pyarrow
side, incompatibility with hadoop version or some settings issue on my
side.
If you have any input I would highly appreciate it.
The python script to upload a file:
import os
> import pyarrow as pa
>
> os.environ["JAVA_HOME"]="<java_home>"
> os.environ['ARROW_LIBHDFS_DIR'] = "<path>/libhdfs.so"
>
> connected = pa.hdfs.connect(host="<host>",port=8020)
>
> destination = "hdfs://<host>:8020/user/tmp/6144m.txt"
> source = "/tmp/6144m.txt"
>
> with connected.open(destination, "wb") as output_stream:
> output_stream.write(open(source, "rb").read())
>
> connected.close()
>
How to create a 6gb file:
truncate -s 6144M 6144m.txt
>
Thanks a lot,
Sergey
Re: Question the nature of the "Zero Copy" advantages of Apache Arrow
Posted by Jorge Cardoso Leitão <jo...@gmail.com>.
Hi Thomas,
The canonical interface that the arrow format offers to share data within
the same process is the C data interface
<https://arrow.apache.org/docs/format/CDataInterface.html>. It offers a
stable ABI to share memory via foreign interfaces. C++, Python, R and
Rust support
it <https://arrow.apache.org/docs/status.html#c-data-interface>.
Best,
Jorge
On Tue, Jan 26, 2021 at 6:47 PM Thomas Browne <th...@crvm.io> wrote:
> So one of the big advantages of Arrow is the common format in memory, on
> the wire, across languages.
>
> I get that this makes it very easy and fast to transfer data between
> nodes, and between languages, which will all share the in-memory format
> and therefore the (often expensive) serialisation step is removed.
>
> However, is it true that one of the core objectives of the project is
> also to allow shared memory objects across different languages on the
> same node? For example, a fast C-based ingest system constantly
> populates a pyarrow buffer, which can be read directly by any other
> application on that node, through pointer sharing?
>
> If this is a core objective, what is the canonical way for brokering the
> "pointers" to this data between languages? Is it the Plasma store? And
> if so, are there plans for Plasma to move be implemented in other client
> languages?
>
> In short. Is Plasma (or if not Plasma, the functionality it provides
> implemented some other way), a core objective of the project?
>
> Or instead is Flight supposed to be used between languages on the same
> node, and if so, does Flight provide true zero-copy (ie - the same
> buffer, not copying the buffer) if run between processes on the same node?
>
> Many thanks.
>
Re: Question the nature of the "Zero Copy" advantages of Apache Arrow
Posted by Daniel Nugent <nu...@gmail.com>.
Right, I recongize that using mmap directly isn't necessarily the most
straightforward, which is why I suggested using a RAM disk with
uncompressed arrow files. It saves the trouble of having to deal with
passing addresses around and puts a nice file system API on top of any
dataset operations that arrow already supports that you might want to do
(you can get a reasonable approach to appends using this, for example).
But if you've already got an on-disk, uncompressed arrow buffer that's
bigger than memory, the arrow api should take care of using the mmap system
calls to load it into memory (at least I think this is currently supported
for all the arrow libraries? You may have to double check. I know it's in
C/C++/Python for sure, probably Rust and I think R?).
Then you're only dealing with virtual allocations and you can load that
larger than memory file in as many analytics packages as you like and there
will only be one copy of any portion of that file in memory at any given
time.
-Dan Nugent
On Tue, Jan 26, 2021 at 1:15 PM Thomas Browne <th...@crvm.io> wrote:
> Yes I think the term "zero copy" was confusing to me. It doesn't quite do
> what it says on the tin since if I understand correctly the term still
> allows for an actually copy still to occur, it's just a direct binary copy
> without a [de]serialisation process.
>
> I hear you on plasma.
>
> On the issue of MAP_SHARED, got it, but that means I'm having to talk C
> from other languages.
>
> I think Jorge's answer (
> https://arrow.apache.org/docs/format/CDataInterface.html) is pretty good
> though. Good enough for me anyway. Thanks everyone.
> On 26/01/2021 18:09, Daniel Nugent wrote:
>
> I think you might be a bit confused about what zero copy means if that’s
> what you’re concerned about. If you have a bigger than memory file, then
> Plasma wasn’t going to help since its design always involved copying the
> arrow buffers to memory.
>
> If you have larger than memory arrow files in the first place, just open
> them using mmap (should be automatically done for non-compressed arrow
> files).
>
> --
> -Dan Nugent
> On Jan 26, 2021, 13:07 -0500, Thomas Browne <th...@crvm.io>
> <th...@crvm.io>, wrote:
>
> don't I lose the benefit of mmapping huge files with a ramdisk? Cos the
> file has to now fit on my ramdisk.
>
> Personally working with financial tick data which can be enormous.
> On 26/01/2021 18:00, Daniel Nugent wrote:
>
> Is there a problem with just using a RAM disk as the method for sharing
> the arrow buffers? It just seems easier and less finicky than a separate
> API to program against.
>
> It also makes storing the data permanently a lot more straightforward, I
> think.
>
> --
> -Dan Nugent
> On Jan 26, 2021, 12:47 -0500, Thomas Browne <th...@crvm.io>
> <th...@crvm.io>, wrote:
>
> So one of the big advantages of Arrow is the common format in memory, on
> the wire, across languages.
>
> I get that this makes it very easy and fast to transfer data between
> nodes, and between languages, which will all share the in-memory format
> and therefore the (often expensive) serialisation step is removed.
>
> However, is it true that one of the core objectives of the project is
> also to allow shared memory objects across different languages on the
> same node? For example, a fast C-based ingest system constantly
> populates a pyarrow buffer, which can be read directly by any other
> application on that node, through pointer sharing?
>
> If this is a core objective, what is the canonical way for brokering the
> "pointers" to this data between languages? Is it the Plasma store? And
> if so, are there plans for Plasma to move be implemented in other client
> languages?
>
> In short. Is Plasma (or if not Plasma, the functionality it provides
> implemented some other way), a core objective of the project?
>
> Or instead is Flight supposed to be used between languages on the same
> node, and if so, does Flight provide true zero-copy (ie - the same
> buffer, not copying the buffer) if run between processes on the same node?
>
> Many thanks.
>
>
Re: Question the nature of the "Zero Copy" advantages of Apache Arrow
Posted by Thomas Browne <th...@crvm.io>.
Yes I think the term "zero copy" was confusing to me. It doesn't quite
do what it says on the tin since if I understand correctly the term
still allows for an actually copy still to occur, it's just a direct
binary copy without a [de]serialisation process.
I hear you on plasma.
On the issue of MAP_SHARED, got it, but that means I'm having to talk C
from other languages.
I think Jorge's answer
(https://arrow.apache.org/docs/format/CDataInterface.html) is pretty
good though. Good enough for me anyway. Thanks everyone.
On 26/01/2021 18:09, Daniel Nugent wrote:
> I think you might be a bit confused about what zero copy means if
> that’s what you’re concerned about. If you have a bigger than memory
> file, then Plasma wasn’t going to help since its design always
> involved copying the arrow buffers to memory.
>
> If you have larger than memory arrow files in the first place, just
> open them using mmap (should be automatically done for non-compressed
> arrow files).
>
> --
> -Dan Nugent
> On Jan 26, 2021, 13:07 -0500, Thomas Browne <th...@crvm.io>, wrote:
>>
>> don't I lose the benefit of mmapping huge files with a ramdisk? Cos
>> the file has to now fit on my ramdisk.
>>
>> Personally working with financial tick data which can be enormous.
>>
>> On 26/01/2021 18:00, Daniel Nugent wrote:
>>> Is there a problem with just using a RAM disk as the method for
>>> sharing the arrow buffers? It just seems easier and less finicky
>>> than a separate API to program against.
>>>
>>> It also makes storing the data permanently a lot more
>>> straightforward, I think.
>>>
>>> --
>>> -Dan Nugent
>>> On Jan 26, 2021, 12:47 -0500, Thomas Browne <th...@crvm.io>, wrote:
>>>> So one of the big advantages of Arrow is the common format in
>>>> memory, on
>>>> the wire, across languages.
>>>>
>>>> I get that this makes it very easy and fast to transfer data between
>>>> nodes, and between languages, which will all share the in-memory format
>>>> and therefore the (often expensive) serialisation step is removed.
>>>>
>>>> However, is it true that one of the core objectives of the project is
>>>> also to allow shared memory objects across different languages on the
>>>> same node? For example, a fast C-based ingest system constantly
>>>> populates a pyarrow buffer, which can be read directly by any other
>>>> application on that node, through pointer sharing?
>>>>
>>>> If this is a core objective, what is the canonical way for
>>>> brokering the
>>>> "pointers" to this data between languages? Is it the Plasma store? And
>>>> if so, are there plans for Plasma to move be implemented in other
>>>> client
>>>> languages?
>>>>
>>>> In short. Is Plasma (or if not Plasma, the functionality it provides
>>>> implemented some other way), a core objective of the project?
>>>>
>>>> Or instead is Flight supposed to be used between languages on the same
>>>> node, and if so, does Flight provide true zero-copy (ie - the same
>>>> buffer, not copying the buffer) if run between processes on the
>>>> same node?
>>>>
>>>> Many thanks.
Re: Question the nature of the "Zero Copy" advantages of Apache
Arrow
Posted by Daniel Nugent <nu...@gmail.com>.
I think you might be a bit confused about what zero copy means if that’s what you’re concerned about. If you have a bigger than memory file, then Plasma wasn’t going to help since its design always involved copying the arrow buffers to memory.
If you have larger than memory arrow files in the first place, just open them using mmap (should be automatically done for non-compressed arrow files).
--
-Dan Nugent
On Jan 26, 2021, 13:07 -0500, Thomas Browne <th...@crvm.io>, wrote:
> don't I lose the benefit of mmapping huge files with a ramdisk? Cos the file has to now fit on my ramdisk.
>
> Personally working with financial tick data which can be enormous.
> On 26/01/2021 18:00, Daniel Nugent wrote:
> > Is there a problem with just using a RAM disk as the method for sharing the arrow buffers? It just seems easier and less finicky than a separate API to program against.
> >
> > It also makes storing the data permanently a lot more straightforward, I think.
> >
> > --
> > -Dan Nugent
> > On Jan 26, 2021, 12:47 -0500, Thomas Browne <th...@crvm.io>, wrote:
> > > So one of the big advantages of Arrow is the common format in memory, on
> > > the wire, across languages.
> > >
> > > I get that this makes it very easy and fast to transfer data between
> > > nodes, and between languages, which will all share the in-memory format
> > > and therefore the (often expensive) serialisation step is removed.
> > >
> > > However, is it true that one of the core objectives of the project is
> > > also to allow shared memory objects across different languages on the
> > > same node? For example, a fast C-based ingest system constantly
> > > populates a pyarrow buffer, which can be read directly by any other
> > > application on that node, through pointer sharing?
> > >
> > > If this is a core objective, what is the canonical way for brokering the
> > > "pointers" to this data between languages? Is it the Plasma store? And
> > > if so, are there plans for Plasma to move be implemented in other client
> > > languages?
> > >
> > > In short. Is Plasma (or if not Plasma, the functionality it provides
> > > implemented some other way), a core objective of the project?
> > >
> > > Or instead is Flight supposed to be used between languages on the same
> > > node, and if so, does Flight provide true zero-copy (ie - the same
> > > buffer, not copying the buffer) if run between processes on the same node?
> > >
> > > Many thanks.
Re: Question the nature of the "Zero Copy" advantages of Apache Arrow
Posted by Thomas Browne <th...@crvm.io>.
don't I lose the benefit of mmapping huge files with a ramdisk? Cos the
file has to now fit on my ramdisk.
Personally working with financial tick data which can be enormous.
On 26/01/2021 18:00, Daniel Nugent wrote:
> Is there a problem with just using a RAM disk as the method for
> sharing the arrow buffers? It just seems easier and less finicky than
> a separate API to program against.
>
> It also makes storing the data permanently a lot more
> straightforward, I think.
>
> --
> -Dan Nugent
> On Jan 26, 2021, 12:47 -0500, Thomas Browne <th...@crvm.io>, wrote:
>> So one of the big advantages of Arrow is the common format in memory, on
>> the wire, across languages.
>>
>> I get that this makes it very easy and fast to transfer data between
>> nodes, and between languages, which will all share the in-memory format
>> and therefore the (often expensive) serialisation step is removed.
>>
>> However, is it true that one of the core objectives of the project is
>> also to allow shared memory objects across different languages on the
>> same node? For example, a fast C-based ingest system constantly
>> populates a pyarrow buffer, which can be read directly by any other
>> application on that node, through pointer sharing?
>>
>> If this is a core objective, what is the canonical way for brokering the
>> "pointers" to this data between languages? Is it the Plasma store? And
>> if so, are there plans for Plasma to move be implemented in other client
>> languages?
>>
>> In short. Is Plasma (or if not Plasma, the functionality it provides
>> implemented some other way), a core objective of the project?
>>
>> Or instead is Flight supposed to be used between languages on the same
>> node, and if so, does Flight provide true zero-copy (ie - the same
>> buffer, not copying the buffer) if run between processes on the same
>> node?
>>
>> Many thanks.
Re: Question the nature of the "Zero Copy" advantages of Apache Arrow
Posted by Chris Nuernberger <ch...@techascent.com>.
Or just mmap with MAP_SHARED
<https://man7.org/linux/man-pages/man2/mmap.2.html>.
On Tue, Jan 26, 2021 at 11:01 AM Daniel Nugent <nu...@gmail.com> wrote:
> Is there a problem with just using a RAM disk as the method for sharing
> the arrow buffers? It just seems easier and less finicky than a separate
> API to program against.
>
> It also makes storing the data permanently a lot more straightforward, I
> think.
>
> --
> -Dan Nugent
> On Jan 26, 2021, 12:47 -0500, Thomas Browne <th...@crvm.io>, wrote:
>
> So one of the big advantages of Arrow is the common format in memory, on
> the wire, across languages.
>
> I get that this makes it very easy and fast to transfer data between
> nodes, and between languages, which will all share the in-memory format
> and therefore the (often expensive) serialisation step is removed.
>
> However, is it true that one of the core objectives of the project is
> also to allow shared memory objects across different languages on the
> same node? For example, a fast C-based ingest system constantly
> populates a pyarrow buffer, which can be read directly by any other
> application on that node, through pointer sharing?
>
> If this is a core objective, what is the canonical way for brokering the
> "pointers" to this data between languages? Is it the Plasma store? And
> if so, are there plans for Plasma to move be implemented in other client
> languages?
>
> In short. Is Plasma (or if not Plasma, the functionality it provides
> implemented some other way), a core objective of the project?
>
> Or instead is Flight supposed to be used between languages on the same
> node, and if so, does Flight provide true zero-copy (ie - the same
> buffer, not copying the buffer) if run between processes on the same node?
>
> Many thanks.
>
>
Re: Question the nature of the "Zero Copy" advantages of Apache
Arrow
Posted by Daniel Nugent <nu...@gmail.com>.
Is there a problem with just using a RAM disk as the method for sharing the arrow buffers? It just seems easier and less finicky than a separate API to program against.
It also makes storing the data permanently a lot more straightforward, I think.
--
-Dan Nugent
On Jan 26, 2021, 12:47 -0500, Thomas Browne <th...@crvm.io>, wrote:
> So one of the big advantages of Arrow is the common format in memory, on
> the wire, across languages.
>
> I get that this makes it very easy and fast to transfer data between
> nodes, and between languages, which will all share the in-memory format
> and therefore the (often expensive) serialisation step is removed.
>
> However, is it true that one of the core objectives of the project is
> also to allow shared memory objects across different languages on the
> same node? For example, a fast C-based ingest system constantly
> populates a pyarrow buffer, which can be read directly by any other
> application on that node, through pointer sharing?
>
> If this is a core objective, what is the canonical way for brokering the
> "pointers" to this data between languages? Is it the Plasma store? And
> if so, are there plans for Plasma to move be implemented in other client
> languages?
>
> In short. Is Plasma (or if not Plasma, the functionality it provides
> implemented some other way), a core objective of the project?
>
> Or instead is Flight supposed to be used between languages on the same
> node, and if so, does Flight provide true zero-copy (ie - the same
> buffer, not copying the buffer) if run between processes on the same node?
>
> Many thanks.
Question the nature of the "Zero Copy" advantages of Apache Arrow
Posted by Thomas Browne <th...@crvm.io>.
So one of the big advantages of Arrow is the common format in memory, on
the wire, across languages.
I get that this makes it very easy and fast to transfer data between
nodes, and between languages, which will all share the in-memory format
and therefore the (often expensive) serialisation step is removed.
However, is it true that one of the core objectives of the project is
also to allow shared memory objects across different languages on the
same node? For example, a fast C-based ingest system constantly
populates a pyarrow buffer, which can be read directly by any other
application on that node, through pointer sharing?
If this is a core objective, what is the canonical way for brokering the
"pointers" to this data between languages? Is it the Plasma store? And
if so, are there plans for Plasma to move be implemented in other client
languages?
In short. Is Plasma (or if not Plasma, the functionality it provides
implemented some other way), a core objective of the project?
Or instead is Flight supposed to be used between languages on the same
node, and if so, does Flight provide true zero-copy (ie - the same
buffer, not copying the buffer) if run between processes on the same node?
Many thanks.
Re: [Python] HDFS write fails when size of file is higher than 6gb
Posted by Wes McKinney <we...@gmail.com>.
It appears that writes over 2GB are implemented incorrectly.
https://github.com/apache/arrow/blob/master/cpp/src/arrow/io/hdfs.cc#L277
the tSize type in libhdfs is an int32_t. So that static cast is truncating data
https://issues.apache.org/jira/browse/ARROW-11391
I would recommend breaking the work into smaller pieces as a workaround
On Tue, Jan 26, 2021 at 1:45 AM Сергей Красовский <kr...@gmail.com> wrote:
>
> Hello Arrow team,
>
> I have an issue with writing files with size > 6143mb to HDFS. Exception is:
>
>> Traceback (most recent call last):
>> File "exp.py", line 22, in <module>
>> output_stream.write(open(source, "rb").read())
>> File "pyarrow/io.pxi", line 283, in pyarrow.lib.NativeFile.write
>> File "pyarrow/error.pxi", line 99, in pyarrow.lib.check_status
>> OSError: HDFS Write failed, errno: 22 (Invalid argument)
>
>
> The code below works for files with size <= 6143mb.
>
> Hadoop version: 3.1.1.3.1.4.0-315
> Python version: 3.6.10
> Pyarrow version: 2.0.0
> System: Ubuntu 16.04.7 LTS
>
> I try to understand what happens under the hood of pyarrow.lib.NativeFile.write. Is there any limitation from pyarrow side, incompatibility with hadoop version or some settings issue on my side.
>
> If you have any input I would highly appreciate it.
>
> The python script to upload a file:
>
>> import os
>> import pyarrow as pa
>>
>> os.environ["JAVA_HOME"]="<java_home>"
>> os.environ['ARROW_LIBHDFS_DIR'] = "<path>/libhdfs.so"
>>
>> connected = pa.hdfs.connect(host="<host>",port=8020)
>>
>> destination = "hdfs://<host>:8020/user/tmp/6144m.txt"
>> source = "/tmp/6144m.txt"
>>
>> with connected.open(destination, "wb") as output_stream:
>> output_stream.write(open(source, "rb").read())
>>
>> connected.close()
>
>
> How to create a 6gb file:
>
>> truncate -s 6144M 6144m.txt
>
>
> Thanks a lot,
> Sergey