You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@arrow.apache.org by we...@apache.org on 2018/01/17 17:26:35 UTC
[arrow] branch master updated: ARROW-2002: [Python] check
write_queue is not full and writer_thread is alive before enqueue new
record when download file.
This is an automated email from the ASF dual-hosted git repository.
wesm pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/arrow.git
The following commit(s) were added to refs/heads/master by this push:
new 9e2fc04 ARROW-2002: [Python] check write_queue is not full and writer_thread is alive before enqueue new record when download file.
9e2fc04 is described below
commit 9e2fc04ddf80d2a04ec789a7ca82c2929d31f967
Author: kmiku7 <ka...@gmail.com>
AuthorDate: Wed Jan 17 12:26:31 2018 -0500
ARROW-2002: [Python] check write_queue is not full and writer_thread is alive before enqueue new record when download file.
use pyarrow download file will raise queue.Full exceptions sometimes.
jira: https://issues.apache.org/jira/browse/ARROW-2002
Author: kmiku7 <ka...@gmail.com>
Closes #1485 from kmiku7/master and squashes the following commits:
8d5f905d [kmiku7] fix queue.FULL exception when writer thread write data slowly.
722182b8 [kmiku7] Merge pull request #1 from apache/master
---
python/pyarrow/io.pxi | 6 ++++++
1 file changed, 6 insertions(+)
diff --git a/python/pyarrow/io.pxi b/python/pyarrow/io.pxi
index 619ba36..5449872 100644
--- a/python/pyarrow/io.pxi
+++ b/python/pyarrow/io.pxi
@@ -312,6 +312,12 @@ cdef class NativeFile:
pybuf = cp.PyBytes_FromStringAndSize(<const char*>buf,
bytes_read)
+ if writer_thread.is_alive():
+ while write_queue.full():
+ time.sleep(0.01)
+ else:
+ break
+
write_queue.put_nowait(pybuf)
finally:
free(buf)
--
To stop receiving notification emails like this one, please contact
['"commits@arrow.apache.org" <co...@arrow.apache.org>'].