You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@beam.apache.org by "Johan Brodin (JIRA)" <ji...@apache.org> on 2017/02/03 15:29:51 UTC

[jira] [Updated] (BEAM-1386) Job hangs without warnings after reading ~20GB of gz csv

     [ https://issues.apache.org/jira/browse/BEAM-1386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Johan Brodin updated BEAM-1386:
-------------------------------
    Priority: Critical  (was: Major)

> Job hangs without warnings after reading ~20GB of gz csv
> --------------------------------------------------------
>
>                 Key: BEAM-1386
>                 URL: https://issues.apache.org/jira/browse/BEAM-1386
>             Project: Beam
>          Issue Type: Bug
>          Components: sdk-py
>    Affects Versions: 0.5.0
>         Environment: Running on Google Dataflow with 'n1-standard-8' machines.
>            Reporter: Johan Brodin
>            Assignee: Ahmet Altay
>            Priority: Critical
>
> When running the job it works fine up until 20GB or around 23 million rows from a gzip:ed csv file (total size 43M rows). Halted the job so the statistic from it seam to disappeared, but here it is the id "2017-02-03_04_25_41-15296331815975218867". Is there any built in limitations to file size? Should I try to break the file up into several smaller files? Could the issue be related to the workers disk size?



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)