You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by "Luke Cwik (JIRA)" <ji...@apache.org> on 2018/03/22 00:08:00 UTC
[jira] [Updated] (BEAM-3908) Leaderboard / gamestats leaking
Dataflow Jobs
[ https://issues.apache.org/jira/browse/BEAM-3908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Luke Cwik updated BEAM-3908:
----------------------------
Description:
I found that the leaderboard/gamestats Dataflow streaming jobs weren't being cleaned up by the test infrastructure which lead to quota issues because all the VMs/disks/memory being consumed causing other Jenkins jobs to fail.
I manually stopped all the jobs that had been running for more then 12 hrs. There were about 20 jobs like this.
Example links:
https://pantheon.corp.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-03-06_22_19_25-7861256924404398606?project=apache-beam-testing&organizationId=433637338589
https://pantheon.corp.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-03-06_18_49_54-7185486205606862436?project=apache-beam-testing&organizationId=433637338589
https://pantheon.corp.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-03-06_23_14_26-6599347078760080693?project=apache-beam-testing&organizationId=433637338589
https://pantheon.corp.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-03-06_18_32_33-7276493109541122240?project=apache-beam-testing&organizationId=433637338589
was:
I found that the leaderboard/gamestats Dataflow streaming jobs weren't being cleaned up by the test infrastructure which lead to quota issues because all the VMs/disks/memory being consumed causing other Jenkins jobs to fail.
I manually stopped all the jobs that had been running for more then 12 hrs.
Example links:
https://pantheon.corp.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-03-06_22_19_25-7861256924404398606?project=apache-beam-testing&organizationId=433637338589
https://pantheon.corp.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-03-06_18_49_54-7185486205606862436?project=apache-beam-testing&organizationId=433637338589
https://pantheon.corp.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-03-06_23_14_26-6599347078760080693?project=apache-beam-testing&organizationId=433637338589
https://pantheon.corp.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-03-06_18_32_33-7276493109541122240?project=apache-beam-testing&organizationId=433637338589
> Leaderboard / gamestats leaking Dataflow Jobs
> ---------------------------------------------
>
> Key: BEAM-3908
> URL: https://issues.apache.org/jira/browse/BEAM-3908
> Project: Beam
> Issue Type: Bug
> Components: runner-dataflow, testing
> Affects Versions: Not applicable
> Reporter: Luke Cwik
> Assignee: Alan Myrvold
> Priority: Critical
>
> I found that the leaderboard/gamestats Dataflow streaming jobs weren't being cleaned up by the test infrastructure which lead to quota issues because all the VMs/disks/memory being consumed causing other Jenkins jobs to fail.
> I manually stopped all the jobs that had been running for more then 12 hrs. There were about 20 jobs like this.
> Example links:
> https://pantheon.corp.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-03-06_22_19_25-7861256924404398606?project=apache-beam-testing&organizationId=433637338589
> https://pantheon.corp.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-03-06_18_49_54-7185486205606862436?project=apache-beam-testing&organizationId=433637338589
> https://pantheon.corp.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-03-06_23_14_26-6599347078760080693?project=apache-beam-testing&organizationId=433637338589
> https://pantheon.corp.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-03-06_18_32_33-7276493109541122240?project=apache-beam-testing&organizationId=433637338589
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)