You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@beam.apache.org by "Eugene Kirpichov (JIRA)" <ji...@apache.org> on 2017/08/27 22:12:00 UTC

[jira] [Created] (BEAM-2810) Consider a faster Avro library in Python

Eugene Kirpichov created BEAM-2810:
--------------------------------------

             Summary: Consider a faster Avro library in Python
                 Key: BEAM-2810
                 URL: https://issues.apache.org/jira/browse/BEAM-2810
             Project: Beam
          Issue Type: Bug
          Components: sdk-py
            Reporter: Eugene Kirpichov
            Assignee: Chamikara Jayalath


https://stackoverflow.com/questions/45870789/bottleneck-on-data-source

Seems like this job is reading Avro files (exported by BigQuery) at about 2 MB/s.

We use the standard Python "avro" library which is apparently known to be very slow (10x+ slower than Java) http://apache-avro.679487.n3.nabble.com/Avro-decode-very-slow-in-Python-td4034422.html, and there are alternatives e.g. https://pypi.python.org/pypi/fastavro/



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)