You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "Lu Qi (JIRA)" <ji...@apache.org> on 2017/11/10 05:22:00 UTC
[jira] [Updated] (ARROW-1792) [Plasma C++] continuous write tensor
failed
[ https://issues.apache.org/jira/browse/ARROW-1792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lu Qi updated ARROW-1792:
--------------------------
Description:
start plasma using "plasma_store -m 8000000000 -s /tmp/plasma"
write tensor in python using
for i in range(10):
client = plasma.connect("/tmp/plasma", "", 0)
x = np.random.rand(1000,1000,5*256).astype("float32") // write 5 GB
object_id = pa.plasma.ObjectID(random_object_id())
tensor = pa.Tensor.from_numpy( x )
data_size = pa.get_tensor_size(tensor)
buf = client.create(object_id, data_size)
stream = pa.FixedSizeBufferWriter(buf)
stream.set_memcopy_threads(6)
pa.write_tensor(tensor, stream)
client.seal(object_id)
// client.release(object_id)
print( i )
The error is like below:
pyarrow.lib.PlasmaStoreFull: object does not fit in the plasma store
If I add "client.release(object_id)" ,the error is:
/arrow/cpp/src/plasma/client.cc296 Check failed: object_entry != objects_in_use_.end()
Also,sometimes error is:
buf = client.create(object_id, data_size)
File "pyarrow/plasma.pyx", line 301, in pyarrow.plasma.PlasmaClient.create (/arrow/python/build/temp.linux-x86_64-2.7/plasma.cxx:4382)
File "pyarrow/error.pxi", line 79, in pyarrow.lib.check_status (/arrow/python/build/temp.linux-x86_64-2.7/lib.cxx:7888)
pyarrow.lib.ArrowIOError: Broken pipe
was:
start plasma using "plasma_store -m 8000000000 -s /tmp/plasma"
write tensor in python using
for i in range(10):
client = plasma.connect("/tmp/plasma", "", 0)
x = np.random.rand(1000,1000,5*256).astype("float32") // write 5 GB
object_id = pa.plasma.ObjectID(random_object_id())
tensor = pa.Tensor.from_numpy(x)
data_size = pa.get_tensor_size(tensor)
buf = client.create(object_id, data_size)
stream = pa.FixedSizeBufferWriter(buf)
stream.set_memcopy_threads(6)
pa.write_tensor(tensor, stream)
client.seal(object_id)
// client.release(object_id)
print(i)
The error is like below:
pyarrow.lib.PlasmaStoreFull: object does not fit in the plasma store
If I add "client.release(object_id)" ,the error is:
/arrow/cpp/src/plasma/client.cc296 Check failed: object_entry != objects_in_use_.end()
Also,sometimes error is:
buf = client.create(object_id, data_size)
File "pyarrow/plasma.pyx", line 301, in pyarrow.plasma.PlasmaClient.create (/arrow/python/build/temp.linux-x86_64-2.7/plasma.cxx:4382)
File "pyarrow/error.pxi", line 79, in pyarrow.lib.check_status (/arrow/python/build/temp.linux-x86_64-2.7/lib.cxx:7888)
pyarrow.lib.ArrowIOError: Broken pipe
> [Plasma C++] continuous write tensor failed
> -------------------------------------------
>
> Key: ARROW-1792
> URL: https://issues.apache.org/jira/browse/ARROW-1792
> Project: Apache Arrow
> Issue Type: Bug
> Components: Plasma (C++)
> Environment: ubuntu 14.04 gcc 4.8.4
> Reporter: Lu Qi
> Original Estimate: 288h
> Remaining Estimate: 288h
>
> start plasma using "plasma_store -m 8000000000 -s /tmp/plasma"
> write tensor in python using
> for i in range(10):
> client = plasma.connect("/tmp/plasma", "", 0)
> x = np.random.rand(1000,1000,5*256).astype("float32") // write 5 GB
> object_id = pa.plasma.ObjectID(random_object_id())
> tensor = pa.Tensor.from_numpy( x )
> data_size = pa.get_tensor_size(tensor)
> buf = client.create(object_id, data_size)
> stream = pa.FixedSizeBufferWriter(buf)
> stream.set_memcopy_threads(6)
> pa.write_tensor(tensor, stream)
> client.seal(object_id)
> // client.release(object_id)
> print( i )
> The error is like below:
> pyarrow.lib.PlasmaStoreFull: object does not fit in the plasma store
> If I add "client.release(object_id)" ,the error is:
> /arrow/cpp/src/plasma/client.cc296 Check failed: object_entry != objects_in_use_.end()
> Also,sometimes error is:
> buf = client.create(object_id, data_size)
> File "pyarrow/plasma.pyx", line 301, in pyarrow.plasma.PlasmaClient.create (/arrow/python/build/temp.linux-x86_64-2.7/plasma.cxx:4382)
> File "pyarrow/error.pxi", line 79, in pyarrow.lib.check_status (/arrow/python/build/temp.linux-x86_64-2.7/lib.cxx:7888)
> pyarrow.lib.ArrowIOError: Broken pipe
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)