You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2018/05/18 14:41:00 UTC
[jira] [Work logged] (BEAM-4041) Performance tests fail due to
kubernetes load balancer problems
[ https://issues.apache.org/jira/browse/BEAM-4041?focusedWorklogId=103393&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-103393 ]
ASF GitHub Bot logged work on BEAM-4041:
----------------------------------------
Author: ASF GitHub Bot
Created on: 18/May/18 14:40
Start Date: 18/May/18 14:40
Worklog Time Spent: 10m
Work Description: DariuszAniszewski opened a new pull request #5425: [BEAM-4041] Increase timeout for getting K8s LoadBalancer external IP
URL: https://github.com/apache/beam/pull/5425
From some time we're observing that obtaining K8s LoadBalancer IP fails, even with 3 minutes timeout. This came less often but still in this week (at least) two jobs suffered this problem:
- `beam_PerformanceTests_MongoDBIO_IT` [build 168](https://builds.apache.org/view/A-D/view/Beam/job/beam_PerformanceTests_MongoDBIO_IT/168/console)
- `beam_PerformanceTests_HadoopInputFormat` [build 256](https://builds.apache.org/view/A-D/view/Beam/job/beam_PerformanceTests_HadoopInputFormat/256/console) and [build 257](https://builds.apache.org/view/A-D/view/Beam/job/beam_PerformanceTests_HadoopInputFormat/257/console)
Since [PR 1641 to PerfkiBenchmarker](https://github.com/GoogleCloudPlatform/PerfKitBenchmarker/pull/1641) it's now possible to configure retry-mechanism policy.
This PR increases overall timeout from 3 minutes to 6 minutes.
----------------
Follow this checklist to help us incorporate your contribution quickly and easily:
- [x] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
It will help us expedite review of your Pull Request if you tag someone (e.g. `@username`) to look at it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 103393)
Time Spent: 10m
Remaining Estimate: 0h
> Performance tests fail due to kubernetes load balancer problems
> ---------------------------------------------------------------
>
> Key: BEAM-4041
> URL: https://issues.apache.org/jira/browse/BEAM-4041
> Project: Beam
> Issue Type: Bug
> Components: testing
> Reporter: Łukasz Gajowy
> Assignee: Jason Kuster
> Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Recently, as we added more IOITs to be run on jenkins using kubernetes, some of them started to fail randomly, because they couldn't retrieve LoadBalancer address. Normally obtaining the address took about one minute. Perfkit waits for the address (actively checking for it) for 3 minutes. This should be enough for getting the address, yet it recently started to exceed the 3 minutes limit. I also noticed that this error didn't happen when there were fewer tests.
> Example logs:
> https://builds.apache.org/view/A-D/view/Beam/job/beam_PerformanceTests_Compressed_TextIOIT_HDFS/31/console
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)