You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2020/06/24 12:10:36 UTC
[GitHub] [beam] damgad opened a new pull request #12072: Adding Jenkins job cleaning the /tmp directory on Jenkins node machine
damgad opened a new pull request #12072:
URL: https://github.com/apache/beam/pull/12072
That PR adds DSL definition of a job that could be used to manually clean /tmp directory on Jenkins nodes. Committers will be able to manually trigger the build and clean the /tmp.
Cleaning is not just simple deleting everything. It deletes only the files which were not accessed during the last 10 hours. That period of time is also configurable. The reason behind that is to make the other executing jobs on the same machine safe.
There is no need to run that job periodically as another, similar mechanism will be introduced as a crontab task on each machine.
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Dataflow | Flink | Samza | Spark
--- | --- | --- | --- | --- | ---
Go | [](https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/)
Java | [](https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/)
Python | [](https://ci-beam.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python35_VR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/)
XLang | --- | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/)
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
--- |Java | Python | Go | Website
--- | --- | --- | --- | ---
Non-portable | [](https://ci-beam.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/)
Portable | --- | [](https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | ---
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aaltay commented on pull request #12072: Adding Jenkins job cleaning the /tmp directory on Jenkins node machine
Posted by GitBox <gi...@apache.org>.
aaltay commented on pull request #12072:
URL: https://github.com/apache/beam/pull/12072#issuecomment-649844411
Let's merge. Could you test it once it is merged?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] damgad commented on pull request #12072: Adding Jenkins job cleaning the /tmp directory on Jenkins node machine
Posted by GitBox <gi...@apache.org>.
damgad commented on pull request #12072:
URL: https://github.com/apache/beam/pull/12072#issuecomment-649844629
Sure, I will :+1:
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aaltay merged pull request #12072: Adding Jenkins job cleaning the /tmp directory on Jenkins node machine
Posted by GitBox <gi...@apache.org>.
aaltay merged pull request #12072:
URL: https://github.com/apache/beam/pull/12072
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] damgad removed a comment on pull request #12072: Adding Jenkins job cleaning the /tmp directory on Jenkins node machine
Posted by GitBox <gi...@apache.org>.
damgad removed a comment on pull request #12072:
URL: https://github.com/apache/beam/pull/12072#issuecomment-648812024
Run RAT PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] damgad commented on pull request #12072: Adding Jenkins job cleaning the /tmp directory on Jenkins node machine
Posted by GitBox <gi...@apache.org>.
damgad commented on pull request #12072:
URL: https://github.com/apache/beam/pull/12072#issuecomment-649843232
@aaltay The cron version (already deployed on the nodes, today) is less strict. It executes once a week and deletes files that were not accessed during at least two days. That should be fine for most cases, and safer for the executing builds.
Here, the mechanism is more powerful, on-demand, and configurable. By specifying the `unaccessed_for` to `0` you can even delete all the tmp files.
I've also considered having only that job with a cron trigger along with the manual one. But that's less reliable. In that case, if the node was for some reason overloaded or builds were stuck the cleanup wouldn't happen.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] damgad commented on pull request #12072: Adding Jenkins job cleaning the /tmp directory on Jenkins node machine
Posted by GitBox <gi...@apache.org>.
damgad commented on pull request #12072:
URL: https://github.com/apache/beam/pull/12072#issuecomment-648828688
Run RAT PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] damgad removed a comment on pull request #12072: Adding Jenkins job cleaning the /tmp directory on Jenkins node machine
Posted by GitBox <gi...@apache.org>.
damgad removed a comment on pull request #12072:
URL: https://github.com/apache/beam/pull/12072#issuecomment-648824559
Run RAT PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] damgad commented on pull request #12072: Adding Jenkins job cleaning the /tmp directory on Jenkins node machine
Posted by GitBox <gi...@apache.org>.
damgad commented on pull request #12072:
URL: https://github.com/apache/beam/pull/12072#issuecomment-648824559
Run RAT PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] damgad commented on pull request #12072: Adding Jenkins job cleaning the /tmp directory on Jenkins node machine
Posted by GitBox <gi...@apache.org>.
damgad commented on pull request #12072:
URL: https://github.com/apache/beam/pull/12072#issuecomment-648786157
R: @aaltay
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] damgad removed a comment on pull request #12072: Adding Jenkins job cleaning the /tmp directory on Jenkins node machine
Posted by GitBox <gi...@apache.org>.
damgad removed a comment on pull request #12072:
URL: https://github.com/apache/beam/pull/12072#issuecomment-648828688
Run RAT PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aaltay commented on pull request #12072: Adding Jenkins job cleaning the /tmp directory on Jenkins node machine
Posted by GitBox <gi...@apache.org>.
aaltay commented on pull request #12072:
URL: https://github.com/apache/beam/pull/12072#issuecomment-649768979
LGTM. Thank you.
One question, if there will be a crontab version, do we need this manual version?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] damgad commented on pull request #12072: Adding Jenkins job cleaning the /tmp directory on Jenkins node machine
Posted by GitBox <gi...@apache.org>.
damgad commented on pull request #12072:
URL: https://github.com/apache/beam/pull/12072#issuecomment-648812024
Run RAT PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] damgad commented on pull request #12072: Adding Jenkins job cleaning the /tmp directory on Jenkins node machine
Posted by GitBox <gi...@apache.org>.
damgad commented on pull request #12072:
URL: https://github.com/apache/beam/pull/12072#issuecomment-649872679
Works like a charm ;) https://ci-beam.apache.org/job/beam_Clean_tmp_directory/3/console
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org