[GitHub] [beam] iemejia commented on issue #11055: [BEAM-9436] Improve GBK in spark structured streaming runner

[GitBox <gi...@apache.org>]

iemejia commented on issue #11055: [BEAM-9436] Improve GBK in spark structured streaming runner
URL: https://github.com/apache/beam/pull/11055#issuecomment-608138463
 
 
   Code looks ok. I could not consistently reproduce the results, when I run the load test in the branch vs master (with the gradleRun patch to include spark-sql) I see results that vary a lot, RAM compsumption is almost equal, but runtime tends to be faster but mostly by a tiny difference on this branch but some runs curiously take 4 more times than others (in both branches), so weird.
   
   I suppose we can merge it in any case if we find some regression (because of the lack of distributed tests we could easily rollback into the previous approach). For ref the command line to run this:
   
   ./gradlew :sdks:java:testing:load-tests:run -Prunner=":runners:spark" -PloadTest.mainClass="org.apache.beam.sdk.loadtests.GroupByKeyLoadTest" -PloadTest.args="--fanout=1 --iterations=1 --streaming=false --runner=SparkStructuredStreamingRunner --sourceOptions={\"numRecords\":200000,\"keySizeBytes\":100000,\"valueSizeBytes\":100000}"
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services