You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by NicoK <gi...@git.apache.org> on 2017/07/10 11:29:32 UTC
[GitHub] flink pull request #4293: [FLINK-7141][build] enable travis cache again
GitHub user NicoK opened a pull request:
https://github.com/apache/flink/pull/4293
[FLINK-7141][build] enable travis cache again
this re-enables the Travis CI cache with a safety net in place that should prevent failing builds because of invalid jar files:
1) we give each job a unique `CACHE_NAME` so that travis should not be the reason for invalid caches (see https://docs.travis-ci.com/user/caching/#Caches-and-build-matrices )
2) we quickly check all jar files in `$HOME/.m2` for invalid ones and remove them (takes around 30s)
additionally, this time, we also remove all our own artefacts from the cache before uploading it
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/NicoK/flink enable_travis_caches
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/4293.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #4293
----
commit 79048712d3342320fdadf16522c4e96352dbf946
Author: Nico Kruber <ni...@data-artisans.com>
Date: 2017-07-06T13:43:43Z
[FLINK-7141][build] enable travis cache again
This adds some additional checks that should make sure the repository may not
get corrupted or if it was than at least it is cleaned up again.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #4293: [FLINK-7141][build] enable travis cache again
Posted by NicoK <gi...@git.apache.org>.
Github user NicoK commented on the issue:
https://github.com/apache/flink/pull/4293
Looks like with the cache it is only downloading flink artefacts from earlier versions for API comparisons (I excluded `$HOME/.m2/repository/org/apache/flink/` from the cache).
Also, it seems there is no real performance gain from using the cache :( or at least the influence of the differences in the runtime of the tests is bigger so that the cache's influence is inconclusive.
With the control over deleting corrupt caches ourselves, we could skip the .m2 repo cleanup and save another 30s there. In the end, we don't know whether corrupt repositories were caused by Travis or Maven - the settings in this PR should fix the former and the repo cleanup the latter. We could thus try removing the latter once we can delete caches ourselves.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #4293: [FLINK-7141][build] enable travis cache again
Posted by zentol <gi...@git.apache.org>.
Github user zentol commented on the issue:
https://github.com/apache/flink/pull/4293
@StephanEwen yes I will take care of this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #4293: [FLINK-7141][build] enable travis cache again
Posted by zentol <gi...@git.apache.org>.
Github user zentol commented on the issue:
https://github.com/apache/flink/pull/4293
yeah I disabled the logging of downloaded libraries, feel free to remove it temporarily.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #4293: [FLINK-7141][build] enable travis cache again
Posted by zentol <gi...@git.apache.org>.
Github user zentol commented on the issue:
https://github.com/apache/flink/pull/4293
Seems like this ain't working for a Pull Request. I also checked your local builds (https://travis-ci.org/NicoK/flink/builds/251943625 & https://travis-ci.org/NicoK/flink/builds/252022358) but it' hard to tell the difference since there still are a lot of timeouts.
I'll give this one a go tomorrow in conjunction with the profile reorganization; those builds don't time out on local builds. Maybe this gives us a better idea how much time we're buying.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #4293: [FLINK-7141][build] enable travis cache again
Posted by NicoK <gi...@git.apache.org>.
Github user NicoK commented on the issue:
https://github.com/apache/flink/pull/4293
that's actually hard to know in advance - I had only 3 builds that went under the limit by chance so I was able to use the cache henceforth in those three which constantly were below the 49min. This PR's Travis build cannot even use the cache yet although it is creating one.
I don't know whether we can simulate this by simply adding another commit on top - by chance I did see something that needs change, so we will see ;)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #4293: [FLINK-7141][build] enable travis cache again
Posted by zentol <gi...@git.apache.org>.
Github user zentol commented on the issue:
https://github.com/apache/flink/pull/4293
Forks also benefit from this.
It would be cool if we knew how much this affects the build times given that we're also adding more work (for the scanning of the cache).
Note that the travis build for the PR has several builds going over the timeout limit.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #4293: [FLINK-7141][build] enable travis cache again
Posted by NicoK <gi...@git.apache.org>.
Github user NicoK commented on the issue:
https://github.com/apache/flink/pull/4293
sounds like a good plan but in the past it wasn't clear where the caches got corrupted, i.e. by travis during cache restore, or by mvn during the build. If the former is still possible, we could only fix it for the next build. Should I try to introducing this technique?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #4293: [FLINK-7141][build] enable travis cache again
Posted by NicoK <gi...@git.apache.org>.
Github user NicoK commented on the issue:
https://github.com/apache/flink/pull/4293
to be honest, I think the cache was used in the second build of the PR as indicated by these lines (of the first build profile):
```
attempting to download cache archive - 3.41s
fetching PR.4293/cache-linux-trusty-cd051817889efc5ff66a50de5c44d5db4ca23a5b661872a975c2c953f95d8ab6--jdk-oraclejdk8.tgz
found cache - 0.00s
adding /home/travis/.m2 to cache - 16.77s
```
Also, there, the cleanup (`test ! -d $HOME/.m2/repository/ || find $HOME/.m2/repository/ -name "*.jar" -exec sh -c 'if ! zip -T {} >/dev/null ; then echo "deleting invalid file: {}"; rm {} ; fi' \;`) took another 36s.
Strangely, in the non-cache build, I do not see any `Downloading: .../maven-metadata.xml` or pom or jar file messages... my guess is that this is due to the parameter `-Dorg.slf4j.simpleLogger.log.org.apache.maven.cli.transfer.Slf4jMavenTransferListener=warn`? - should we disable it for testing that there are actually no new downloads with the cache in place?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #4293: [FLINK-7141][build] enable travis cache again
Posted by zentol <gi...@git.apache.org>.
Github user zentol commented on the issue:
https://github.com/apache/flink/pull/4293
Ok, now i agree that we should enable the cache as well.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #4293: [FLINK-7141][build] enable travis cache again
Posted by greghogan <gi...@git.apache.org>.
Github user greghogan commented on the issue:
https://github.com/apache/flink/pull/4293
Not only timeouts but we're likely downloading multiple gigabytes across the 12 builds per TravisCI job.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #4293: [FLINK-7141][build] enable travis cache again
Posted by NicoK <gi...@git.apache.org>.
Github user NicoK commented on the issue:
https://github.com/apache/flink/pull/4293
Once we are able to delete the travis cache on our own, we may experiment a bit more and try without the cleanup task or just running it for failed jobs.
For now, though, @StephanEwen has a point about activating the cache anyway: getting rid of the artefact download timeouts.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #4293: [FLINK-7141][build] enable travis cache again
Posted by zentol <gi...@git.apache.org>.
Github user zentol commented on the issue:
https://github.com/apache/flink/pull/4293
merging.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #4293: [FLINK-7141][build] enable travis cache again
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/4293
@zentol Do you want to merge and validate this this as part of your ongoing build optimization project?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #4293: [FLINK-7141][build] enable travis cache again
Posted by zentol <gi...@git.apache.org>.
Github user zentol commented on the issue:
https://github.com/apache/flink/pull/4293
Given that enabling the cache doesn't appear to improve build times I wouldn't enable it, especially since it does require more efforts for cleaning up corrupted caches. It appears there are only downsides for us to enabling it.
If we don't cache artifacts across job we don't have to worry about fixing corrupted artifacts, so that problem would be "solved".
The check is still useful though.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #4293: [FLINK-7141][build] enable travis cache again
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/flink/pull/4293
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #4293: [FLINK-7141][build] enable travis cache again
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/4293
I kind of like having automatic checks for corrupt caches. The 30s should not be a big issue.
One thing that this may give us is more stability against failed dependency downloads, which does happen not too infrequently. That would actually be nice!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #4293: [FLINK-7141][build] enable travis cache again
Posted by zentol <gi...@git.apache.org>.
Github user zentol commented on the issue:
https://github.com/apache/flink/pull/4293
We only have to check for corrupted artifacts if the build actually failed; if it succeeded we implicitly know that the artifacts are OK. This gives us the debugging benefits, without increasing the build times.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---