[jira] [Work logged] (BEAM-9436) Improve performance of GBK

["ASF GitHub Bot (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/BEAM-9436?focusedWorklogId=415120&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-415120 ]

ASF GitHub Bot logged work on BEAM-9436:
----------------------------------------

                Author: ASF GitHub Bot
            Created on: 02/Apr/20 23:11
            Start Date: 02/Apr/20 23:11
    Worklog Time Spent: 10m 
      Work Description: iemejia commented on issue #11055: [BEAM-9436] Improve GBK in spark structured streaming runner
URL: https://github.com/apache/beam/pull/11055#issuecomment-608138463
 
 
   Code looks ok. I could not consistently reproduce the results, when I run the load test in the branch vs master (with the gradleRun patch to include spark-sql) I see results that vary a lot, RAM compsumption is almost equal, but runtime tends to be faster but mostly by a tiny difference on this branch but some runs curiously take 4 more times than others (in both branches), so weird.
   
   I suppose we can merge it in any case if we find some regression (because of the lack of distributed tests we could easily rollback into the previous approach). For ref the command line to run this:
   
   ./gradlew :sdks:java:testing:load-tests:run -Prunner=":runners:spark" -PloadTest.mainClass="org.apache.beam.sdk.loadtests.GroupByKeyLoadTest" -PloadTest.args="--fanout=1 --iterations=1 --streaming=false --runner=SparkStructuredStreamingRunner --sourceOptions={\"numRecords\":200000,\"keySizeBytes\":100000,\"valueSizeBytes\":100000}"
   
 
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Issue Time Tracking
-------------------

    Worklog Id:     (was: 415120)
    Time Spent: 14h 50m  (was: 14h 40m)

> Improve performance of GBK
> --------------------------
>
>                 Key: BEAM-9436
>                 URL: https://issues.apache.org/jira/browse/BEAM-9436
>             Project: Beam
>          Issue Type: Improvement
>          Components: runner-spark
>            Reporter: Etienne Chauchot
>            Assignee: Etienne Chauchot
>            Priority: Major
>              Labels: structured-streaming
>          Time Spent: 14h 50m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)