[jira] [Commented] (BEAM-2810) Consider a faster Avro library in Python

["Chamikara Jayalath (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/BEAM-2810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16143293#comment-16143293 ] 

Chamikara Jayalath commented on BEAM-2810:
------------------------------------------

I believe we considered fastavro but couldn't use it since it does not expose the byte position when reading. Not sure if the API has changed since.

> Consider a faster Avro library in Python
> ----------------------------------------
>
>                 Key: BEAM-2810
>                 URL: https://issues.apache.org/jira/browse/BEAM-2810
>             Project: Beam
>          Issue Type: Bug
>          Components: sdk-py
>            Reporter: Eugene Kirpichov
>            Assignee: Chamikara Jayalath
>
> https://stackoverflow.com/questions/45870789/bottleneck-on-data-source
> Seems like this job is reading Avro files (exported by BigQuery) at about 2 MB/s.
> We use the standard Python "avro" library which is apparently known to be very slow (10x+ slower than Java) http://apache-avro.679487.n3.nabble.com/Avro-decode-very-slow-in-Python-td4034422.html, and there are alternatives e.g. https://pypi.python.org/pypi/fastavro/



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)