You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by "Anant Bhandarkar (JIRA)" <ji...@apache.org> on 2017/05/09 06:22:04 UTC
[jira] [Comment Edited] (BEAM-2208) Python SDK wordcount on cloud
Dataflow runner is slow
[ https://issues.apache.org/jira/browse/BEAM-2208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16002132#comment-16002132 ]
Anant Bhandarkar edited comment on BEAM-2208 at 5/9/17 6:21 AM:
----------------------------------------------------------------
[~altay] This word count job was run yesterday.
2017-05-08_02_48_51-5929018952297525369
We tried to increase the number of worker instance to 50 instead of autoscale but it only took max 2 workers and took 34 min 54 sec to execute.
Wondering what will ensure that the work is distributed among the workers also what will bring about such difference in execution times compared to Java in a simple word count scenario.
was (Author: anantmb@gmail.com):
[~altay] This word count job was run yesterday.
2017-05-08_02_48_51-5929018952297525369
We tried to increase the number of worker instance to 50 instead of autoscale but it only took max 2 workers and took 34 min 54 sec to execute.
Wondering what will ensure that the work is distributed among the workers also what will bring about such difference in execution times compared to Java in a word count scenario.
> Python SDK wordcount on cloud Dataflow runner is slow
> -----------------------------------------------------
>
> Key: BEAM-2208
> URL: https://issues.apache.org/jira/browse/BEAM-2208
> Project: Beam
> Issue Type: Improvement
> Components: runner-dataflow, sdk-py
> Affects Versions: 0.6.0
> Reporter: Anant Bhandarkar
> Assignee: Ahmet Altay
> Priority: Critical
>
> I have been trying to run the Beam Word count example with a 2GB file.
> When I run the Java Example for word count of this csv file the job gets completed in 7.15secs Mins.
> Job ID
> 2017-04-18_23_57_02-2832613177376293063
> But word count example with same file using Python SDK takes 28 to 35mins 2017-04-20_04_48_27-8924552896141769408
> SDK version
> Apache Beam SDK for Python 0.6.0
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)