You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "Wes McKinney (JIRA)" <ji...@apache.org> on 2018/09/15 14:56:01 UTC
[jira] [Commented] (ARROW-2879) [Python] Arrow plasma can only use
a small part of specified shared memory
[ https://issues.apache.org/jira/browse/ARROW-2879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16616342#comment-16616342 ]
Wes McKinney commented on ARROW-2879:
-------------------------------------
What is the status here?
> [Python] Arrow plasma can only use a small part of specified shared memory
> --------------------------------------------------------------------------
>
> Key: ARROW-2879
> URL: https://issues.apache.org/jira/browse/ARROW-2879
> Project: Apache Arrow
> Issue Type: Bug
> Components: Python
> Reporter: chineking
> Priority: Major
> Fix For: 0.12.0
>
>
> Hi, thanks for the great job of arrow, it helps us a lot.
> However, we encounter a problem when we were using plasma.
> The sample code:
> {code:python}
> import numpy as np
> import pyarrow as pa
> import pyarrow.plasma as plasma
> client = plasma.connect("/tmp/plasma", "", 0)
> puts = []
> nbytes = 0
> while True:
> a = np.ones((1000, 1000))
> try:
> oid = client.put(a)
> puts.append(client.get(oid))
> nbytes += a.nbytes
> except pa.lib.PlasmaStoreFull:
> print('use nbytes', nbytes)
> break
> {code}
> We start a plasma store with 1G memory, but the nbytes output above is only 496000000, which cannot even reach half of the memory we specified.
> I cannot figure out why plasma can only use such a small part of shared memory. Could anybody help me? Thanks a lot.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)