[ https://issues.apache.org/jira/browse/BEAM-6291?focusedWorklogId=189555&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-189555 ]
ASF GitHub Bot logged work on BEAM-6291:
----------------------------------------
Author: ASF GitHub Bot
Created on: 24/Jan/19 16:51
Start Date: 24/Jan/19 16:51
Worklog Time Spent: 10m
Work Description: kkucharc commented on pull request #7614: [BEAM-6291] Generic BigQuery schema load tests metrics
URL: https://github.com/apache/beam/pull/7614
It was decided to change tables to have `metrics` column where it will be name of metric which is collected.
Also added two minor changes:
* added an env variable to disable load tests,
* added pipeline option to disable saving metrics
------------------------
Follow this checklist to help us incorporate your contribution quickly and easily:
- [x] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
It will help us expedite review of your Pull Request if you tag someone (e.g. `@username`) to look at it.
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
--- | --- | --- | --- | --- | --- | --- | ---
Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | --- | --- | --- | ---
Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)
Python | [](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/) </br> [](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | --- | --- | --- | ---
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 189555)
Time Spent: 10m
Remaining Estimate: 0h
> Make the schema for BQ tables storing metric results more generic (Python)
> --------------------------------------------------------------------------
>
> Key: BEAM-6291
> URL: https://issues.apache.org/jira/browse/BEAM-6291
> Project: Beam
> Issue Type: Sub-task
> Components: testing
> Reporter: Lukasz Gajowy
> Assignee: Kasia Kucharczyk
> Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Currently, we keep the metrics results in BQ in tables with a schema like this:
> timestamp | total_bytes | run_time | (possibly other BQ columns)
> every time we want to add a new column the schema has to be extended. This is not convenient given the fact that any load test can have different metrics stored. This in turn would cause multiple BQ tables each queried differently.
> We can provide a more generic schema, like so:
> test_id | timestamp | metric | value
> thanks to that, every metric, whatever it's name is, can be saved in the table as a separate row. This gives more elasticity in storing metrics and is still easy to query and plot.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)