You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by "Ryan Williams (JIRA)" <ji...@apache.org> on 2018/07/02 18:30:00 UTC
[jira] [Commented] (BEAM-2810) Consider a faster Avro library in
Python
[ https://issues.apache.org/jira/browse/BEAM-2810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16530287#comment-16530287 ]
Ryan Williams commented on BEAM-2810:
-------------------------------------
This is basically done since [#5496|https://github.com/apache/beam/pull/5496] is merged: the various avroio {{PTransforms}} all take a {{use_fastavro}} param (default: {{False}}).
I'm going to finish adding [the integration test that I took out|https://github.com/apache/beam/pull/5496#issuecomment-399954425], so we can leave this open to cover that work as well if that makes the most sense.
> Consider a faster Avro library in Python
> ----------------------------------------
>
> Key: BEAM-2810
> URL: https://issues.apache.org/jira/browse/BEAM-2810
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
> Reporter: Eugene Kirpichov
> Assignee: Ryan Williams
> Priority: Major
> Time Spent: 5h
> Remaining Estimate: 0h
>
> https://stackoverflow.com/questions/45870789/bottleneck-on-data-source
> Seems like this job is reading Avro files (exported by BigQuery) at about 2 MB/s.
> We use the standard Python "avro" library which is apparently known to be very slow (10x+ slower than Java) http://apache-avro.679487.n3.nabble.com/Avro-decode-very-slow-in-Python-td4034422.html, and there are alternatives e.g. https://pypi.python.org/pypi/fastavro/
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)