You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2018/10/04 15:35:00 UTC
[jira] [Work logged] (BEAM-5355) Create GroupByKey load test for
Java SDK
[ https://issues.apache.org/jira/browse/BEAM-5355?focusedWorklogId=151230&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-151230 ]
ASF GitHub Bot logged work on BEAM-5355:
----------------------------------------
Author: ASF GitHub Bot
Created on: 04/Oct/18 15:34
Start Date: 04/Oct/18 15:34
Worklog Time Spent: 10m
Work Description: lgajowy opened a new pull request #6571: [BEAM-5355] Add GroupByKeyLoadTest
URL: https://github.com/apache/beam/pull/6571
This commit adds the test with all neecssary setup required to run itmanually.
The following GroupByKey load test can be run on all runners using the following command (example is for flink):
```
./gradlew run -p sdks/java/load-tests -PloadTest.args='--fanout=5 --iterations=1 --sourceOptions={"numRecords":1000,"splitPointFrequencyRecords":1,"keySizeBytes":10,"valueSizeBytes":20,"numHotKeys":3,"hotKeyFraction":0.3,"seed":123456,"bundleSizeDistribution":{"type":"const","const":42},"forceNumInitialBundles":100,"progressShape":"LINEAR","initializeDelayDistribution":{"type":"const","const":42}} --stepOptions={"outputRecordsPerInputRecord":2,"preservesInputKeyDistribution":2,"keySizeBytes":10,"valueSizeBytes":20,"numHotKeys":3,"hotKeyFraction":0.3,"seed":123456}} --runner=FlinkRunner' -Prunner=":beam-runners-flink_2.11" -PloadTest.mainClass="org.apache.beam.sdk.loadtests.GroupByKeyLoadTest"
```
Runner can be changed by altering the runner-related properties. We can also add `--flinkMaster=localhost:8081` to run this on a separate flink cluster (not in memory).
You can skip the syntieticStep pipeline option. Example for dataflow runner:
```
./gradlew run -p sdks/java/load-tests -PloadTest.args='--fanout=5 --iterations=1 --sourceOptions={"numRecords":1000,"splitPointFrequencyRecords":1,"keySizeBytes":10,"valueSizeBytes":20,"numHotKeys":3,"hotKeyFraction":0.3,"seed":123456,"bundleSizeDistribution":{"type":"const","const":42},"forceNumInitialBundles":100,"progressShape":"LINEAR","initializeDelayDistribution":{"type":"const","const":42}} --runner=DataflowRunner' -Prunner=":beam-runners-google-cloud-dataflow-java" -PloadTest.mainClass="org.apache.beam.loadtests.GroupByKeyLoadTest"
```
@pabloem could you take a look? I know there is a separate PR for this but it was easier to completely rebuild the pr due to the fact that now it does not require Perfkit and runs from the main code. I didn't develop any method of publishing the metrics to BigQuery - this needs a follow up pull request (is it ok?).
CC: @echauchot
------------------------
Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
It will help us expedite review of your Pull Request if you tag someone (e.g. `@username`) to look at it.
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
--- | --- | --- | --- | --- | --- | --- | ---
Go | [](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/) | --- | --- | --- | --- | --- | ---
Java | [](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/)
Python | [](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/) </br> [](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/) | --- | --- | ---
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 151230)
Time Spent: 1h (was: 50m)
> Create GroupByKey load test for Java SDK
> ----------------------------------------
>
> Key: BEAM-5355
> URL: https://issues.apache.org/jira/browse/BEAM-5355
> Project: Beam
> Issue Type: New Feature
> Components: testing
> Reporter: Lukasz Gajowy
> Assignee: Lukasz Gajowy
> Priority: Minor
> Time Spent: 1h
> Remaining Estimate: 0h
>
> This is more thoroughly described in this proposal: [https://docs.google.com/document/d/1PuIQv4v06eosKKwT76u7S6IP88AnXhTf870Rcj1AHt4/edit?usp=sharing]
> In short: this ticket is about implementing the GroupByKeyLoadIT that uses SyntheticStep and Synthetic source to create load on the pipeline.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)