[jira] [Comment Edited] (ARROW-723) Arrow freezes on write if chunk_size=0

["Jonathan Chambers (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ARROW-723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15950610#comment-15950610 ] 

Jonathan Chambers edited comment on ARROW-723 at 3/31/17 9:45 AM:
------------------------------------------------------------------

I don't have time to explore this much unfortunately, but I can share a few observations:

- Running on Anaconda Python 3.6  in conda env on macOS 10.12.3
- When run in an interactive console, this bug completely freezes the process - cannot exit with ctrl-c etc, need to kill the process.
- When run through Pycharm's interactive debugger, it gets to the write file line and then freezes - no way to continue debugging, forced to kill debugger.
- A test.pq file is written that seems to increase in size indefinitely if you let the process run.
- CPU usage is 100%
- These suggest that pyarrow get's stuck in an infinite loop somewhere in the C part of the code.

Hope that helps


was (Author: mangecoeur):
I don't have time to explore this much unfortunately, but I can share a few observations:

- When run in an interactive console, this bug completely freezes the process - cannot exit with ctrl-c etc, need to kill the process.
- When run through Pycharm's interactive debugger, it gets to the write file line and then freezes - no way to continue debugging, forced to kill debugger.
- A test.pq file is written that seems to increase in size indefinitely if you let the process run.
- CPU usage is 100%
- These suggest that pyarrow get's stuck in an infinite loop somewhere in the C part of the code.

Hope that helps

> Arrow freezes on write if chunk_size=0
> --------------------------------------
>
>                 Key: ARROW-723
>                 URL: https://issues.apache.org/jira/browse/ARROW-723
>             Project: Apache Arrow
>          Issue Type: Bug
>          Components: Python
>    Affects Versions: 0.2.0
>         Environment: Linux, macOS
>            Reporter: Jonathan Chambers
>
> Pyarrow freezes if you set chunk_size=0 (e.g. if you forget to account for short data when setting chunk size as a function of table length, see example).
> Would expect either to handle gracefully (e.g. revert to behaviour chunk_size=None) or to throw error.
> ```
> import numpy as np
> import pandas as pd
> import pyarrow as pa
> import pyarrow.parquet as pq
> cols = 'A', 'B', 'C', 'D'
> row = np.arange(4)
> data = pd.DataFrame([row], columns=cols)
> table = pa.Table.from_pandas(data.reset_index(), timestamps_to_ms=True)
> pq.write_table(table, 'test.pq', chunk_size=int(len(data) / 4))
> ```



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)