You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by uce <gi...@git.apache.org> on 2015/02/16 17:34:18 UTC
[GitHub] flink pull request: [builds] Allow to upload build artifacts to S3
GitHub user uce opened a pull request:
https://github.com/apache/flink/pull/407
[builds] Allow to upload build artifacts to S3
This PR includes #389.
Currently, logging of Travis builds is disabled as the log files are discarded after a build and stdout logging is limited in size. With this PR, we can log to a file and have them uploaded automatically.
1. **Console logging**: For console logging, we can use `MavenForkNumberPrefixLayout` to prefix the log output with the fork number, which makes it easier to group log output together.
We might consider moving that layout to a different package, though.
```
7 > 17:12:23,220 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying test vertex (5/7) (attempt #0) to host3
7 > 17:12:23,220 DEBUG org.apache.flink.runtime.jobmanager.scheduler.Scheduler - Scheduling task {task=test vertex (6/7) - execution #0, sharingUnit=null, locationConstraint=null}
7 > 17:12:23,220 DEBUG org.apache.flink.runtime.jobmanager.scheduler.Scheduler - Unconstrained assignment: test vertex (6/7) --> dbc442af2610e30d3309d96e14fa9d8a @ akka://TestingActorSystem/user/$$a - 5 slots - 1067599825
7 > 17:12:23,221 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying test vertex (6/7) (attempt #0) to host1
7 > 17:12:23,221 DEBUG org.apache.flink.runtime.jobmanager.scheduler.Scheduler - Scheduling task {task=test vertex (7/7) - execution #0, sharingUnit=null, locationConstraint=null}
7 > 17:12:23,222 DEBUG org.apache.flink.runtime.jobmanager.scheduler.Scheduler - Unconstrained assignment: test vertex (7/7) --> 8ce93c214951df5a8ad32c912ba0e456 @ akka://TestingActorSystem/user/$$a - 5 slots - 749927456
7 > 17:12:23,222 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying test vertex (7/7) (attempt #0) to host2
7 > 17:12:23,223 DEBUG org.apache.flink.runtime.executiongraph.ExecutionGraph - Attaching 1 topologically sorted vertices to existing job graph with 0 vertices and 0 intermediate results.
7 > 17:12:23,223 DEBUG org.apache.flink.runtime.executiongraph.ExecutionGraph - Connecting ExecutionJobVertex 126e16c348d4e5aa129104702700bf9e (test vertex) to 0 predecessors.
10 > 17:12:23,225 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying v1 (2/2) (attempt #0) to host3
6 > 17:12:23,236 DEBUG akka.event.EventStream - shutting down: StandardOutLogger started
1 > 17:12:23,239 DEBUG org.apache.flink.runtime.executiongraph.ExecutionGraph - Sending out cancel request, to remove task execution from TaskManager.
1 > 17:12:23,244 DEBUG org.apache.flink.runtime.executiongraph.ExecutionGraph - Concurrent canceling/failing of TestVertex (1/1) - execution #0 while deployment was in progress.
10 > 17:12:23,266 DEBUG org.apache.flink.runtime.executiongraph.ExecutionGraph - Attaching 1 topologically sorted vertices to existing job graph with 0 vertices and 0 intermediate results.
Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.61 sec - in org.apache.flink.runtime.executiongraph.ExecutionVertexCancelTest
Running org.apache.flink.runtime.io.network.api.reader.UnionBufferReaderTest
10 > 17:12:23,269 DEBUG org.apache.flink.runtime.executiongraph.ExecutionGraph - Connecting ExecutionJobVertex e8869426e7765e498ad219e3af731729 (test vertex) to 0 predecessors.
10 > 17:12:23,269 DEBUG org.apache.flink.runtime.jobmanager.scheduler.Scheduler - Scheduling task {task=test vertex (1/2) - execution #0, sharingUnit=null, locationConstraint=null}
10 > 17:12:23,270 DEBUG org.apache.flink.runtime.jobmanager.scheduler.Scheduler - Local assignment: test vertex (1/2) --> ee9357ffc16d51c4f7522bc479a757ac @ akka://TestingActorSystem/user/$$a - 1 slots - 2073640037
10 > 17:12:23,270 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying test vertex (1/2) (attempt #0) to host1
10 > 17:12:23,270 DEBUG org.apache.flink.runtime.jobmanager.scheduler.Scheduler - Scheduling task {task=test vertex (2/2) - execution #0, sharingUnit=null, locationConstraint=null}
10 > 17:12:23,271 DEBUG org.apache.flink.runtime.jobmanager.scheduler.Scheduler - Local assignment: test vertex (2/2) --> 6620d9f45b7567e2b147f5761d8060e8 @ akka://TestingActorSystem/user/$$a - 1 slots - 1864116663
10 > 17:12:23,271 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying test vertex (2/2) (attempt #0) to host3
1 > 17:12:23,277 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying TestVertex (1/1) (attempt #0) to localhost
10 > 17:12:23,277 DEBUG org.apache.flink.runtime.executiongraph.ExecutionGraph - Attaching 1 topologically sorted vertices to existing job graph with 0 vertices and 0 intermediate results.
10 > 17:12:23,278 DEBUG org.apache.flink.runtime.executiongraph.ExecutionGraph - Connecting ExecutionJobVertex 533fcd1e05bcce9ec185ccea7085a5aa (test vertex) to 0 predecessors.
10 > 17:12:23,278 DEBUG org.apache.flink.runtime.jobmanager.scheduler.Scheduler - Scheduling task {task=test vertex (1/2) - execution #0, sharingUnit=null, locationConstraint=null}
1 > 17:12:23,280 DEBUG org.apache.flink.runtime.executiongraph.ExecutionGraph - Sending out cancel request, to remove task execution from TaskManager.
1 > 17:12:23,280 DEBUG org.apache.flink.runtime.executiongraph.ExecutionGraph - Cancel task call did not find task. Probably akka message call race.
1 > 17:12:23,281 DEBUG org.apache.flink.runtime.executiongraph.ExecutionGraph - Concurrent canceling/failing of TestVertex (1/1) - execution #0 while deployment was in progress.
10 > 17:12:23,281 DEBUG org.apache.flink.runtime.jobmanager.scheduler.Scheduler - Local assignment: test vertex (1/2) --> b923679116189038c5dbd280d43cfe48 @ akka://TestingActorSystem/user/$$a - 1 slots - 44559647
10 > 17:12:23,282 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying test vertex (1/2) (attempt #0) to host3
10 > 17:12:23,282 DEBUG org.apache.flink.runtime.jobmanager.scheduler.Scheduler - Scheduling task {task=test vertex (2/2) - execution #0, sharingUnit=null, locationConstraint=null}
10 > 17:12:23,283 DEBUG org.apache.flink.runtime.jobmanager.scheduler.Scheduler - Local assignment: test vertex (2/2) --> 1fb78d024e2664602ee31c47dd987f6c @ akka://TestingActorSystem/user/$$a - 1 slots - 1081769770
```
I've added an example log4j properties file to `flink-runtime/tests`.
2. **File logging**: Each forked build sets its own log file (e.g. `1.log` and `2.log`) if it is enabled (example in `flink-runtime/tests` log4j properties file). If the `UPLOAD_ARTIFACTS` variable is set in Travis and the S3 credentials have been set for the respective repository, the files are uploaded.
As an example, see [this build](https://travis-ci.org/uce/incubator-flink/builds/50960309), which I've stalled on purpose. It upload the jps traces: https://s3.amazonaws.com/flink.a.o.uce.east/travis-artifacts/uce/incubator-flink/210/210.5/jps-traces.out
---
The used S3 bucket is my private bucket and the upload currently only works for my forked repository. It is very easy to add more committers for their own repositories/own buckets.
We can finalize the setup for interested committers after review. I still want to add a minor change to compress the output.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/uce/incubator-flink travis-upload_artifacts
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/407.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #407
----
commit 7887599e27949f72373232205d15010c18108bc2
Author: Ufuk Celebi <uc...@apache.org>
Date: 2015-02-12T11:19:40Z
[builds] Print Java process stack traces of stalled builds
commit cb17e87b3d47336f50999c2e239d8ec6467e0a59
Author: Ufuk Celebi <uc...@apache.org>
Date: 2015-02-16T15:40:49Z
[builds] Add logger layout to prefix log output with Maven fork number
commit 488946ee273de9c374ec0a87652077c355718cae
Author: Ufuk Celebi <uc...@apache.org>
Date: 2015-02-16T15:42:13Z
[builds] Move .gitignores and set JVM properties when running build
commit 2a942e0980e80201acc954abc20401a3f9c8764c
Author: Ufuk Celebi <uc...@apache.org>
Date: 2015-02-16T15:48:07Z
[builds] Allow to upload build artifacts to S3
commit 5cd3e395cea2e64b76edebb26ec6a517b111fc85
Author: Ufuk Celebi <uc...@apache.org>
Date: 2015-02-16T15:56:48Z
[builds] Fix rat excludes
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [builds] Allow to upload build artifacts to S3
Posted by uce <gi...@git.apache.org>.
Github user uce commented on the pull request:
https://github.com/apache/flink/pull/407#issuecomment-75035273
OK, rebased and fixed a remaining issue with the help of @rmetzger.
With the current version, all builds of apache/flink will be logged to file and uploaded to S3.
Ping me, if you want this enabled for single committer repos (currently it's my S3 bucket).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [builds] Allow to upload build artifacts to S3
Posted by uce <gi...@git.apache.org>.
Github user uce commented on the pull request:
https://github.com/apache/flink/pull/407#issuecomment-75038691
Added some documentation here: https://cwiki.apache.org/confluence/display/FLINK/Travis+log+output
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [builds] Allow to upload build artifacts to S3
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on the pull request:
https://github.com/apache/flink/pull/407#issuecomment-75049005
Great Ufuk. Could you enable it for my repo?
On Thu, Feb 19, 2015 at 12:42 PM, Ufuk Celebi <no...@github.com>
wrote:
> Added some documentation here:
> https://cwiki.apache.org/confluence/display/FLINK/Travis+log+output
>
> —
> Reply to this email directly or view it on GitHub
> <https://github.com/apache/flink/pull/407#issuecomment-75038691>.
>
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [builds] Allow to upload build artifacts to S3
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on the pull request:
https://github.com/apache/flink/pull/407#issuecomment-75018451
Much needed feature.
+1
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [builds] Allow to upload build artifacts to S3
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on the pull request:
https://github.com/apache/flink/pull/407#issuecomment-75049700
Nice, thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [builds] Allow to upload build artifacts to S3
Posted by uce <gi...@git.apache.org>.
Github user uce commented on the pull request:
https://github.com/apache/flink/pull/407#issuecomment-75039103
Should I add some committer fork repos to the configuration to allow uploading of log files there as well? Currently, it only uploads logs of builds for apache/flink.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [builds] Allow to upload build artifacts to S3
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on the pull request:
https://github.com/apache/flink/pull/407#issuecomment-75055735
Thanks Ufuk. That is really helpful.
On Thu, Feb 19, 2015 at 2:14 PM, Stephan Ewen <no...@github.com>
wrote:
> Nice, thanks!
>
> —
> Reply to this email directly or view it on GitHub
> <https://github.com/apache/flink/pull/407#issuecomment-75049700>.
>
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [builds] Allow to upload build artifacts to S3
Posted by uce <gi...@git.apache.org>.
Github user uce commented on the pull request:
https://github.com/apache/flink/pull/407#issuecomment-75049539
OK, I've set it to our main repo and the forks of: @tillrohrmann, @rmetzger, @StephanEwen, and my own.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [builds] Allow to upload build artifacts to S3
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/flink/pull/407
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [builds] Allow to upload build artifacts to S3
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on the pull request:
https://github.com/apache/flink/pull/407#issuecomment-74864911
Very good idea.
+1 from my side
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---