You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@trafficcontrol.apache.org by "Fieck, Brennan" <Br...@comcast.com> on 2019/01/24 23:13:20 UTC
Re: [EXTERNAL] CiaB Test Compose Outputting JUnit?
I'm +1. If nothing else, it doesn't add any dependencies to actually *running* ATC or CIAB,
and JUnit files are technically human-readable (though I certainly wouldn't love doing that
often)>
________________________________________
From: Robert Butts <ro...@apache.org>
Sent: Thursday, January 24, 2019 2:35 PM
To: dev@trafficcontrol.apache.org
Subject: [EXTERNAL] CiaB Test Compose Outputting JUnit?
Does anyone object to making the cdn-in-a-box test compose files (e.g.
https://github.com/apache/trafficcontrol/blob/master/infrastructure/cdn-in-a-box/docker-compose.traffic-ops-test.yml)
output JUnit format?
I made an internal fork to do that for
`docker-compose.traffic-ops-test.yml` as well as complete
Dockerfiles+Compose for the TM, Portal, and Router tests. At the time, I
didn't OSS them, because I thought we shouldn't impose JUnit on TC.
But in retrospect, it limits our contributions to TC for no good reason.
Further, TC uses the Apache Jenkins server, which expects JUnit. So I'm not
seeing a reason not to standardize all of our docker-compose test
infrastructure around JUnit.
Note I'm only talking about the `docker-compose` infrastructure to run
tests, not the tests themselves. The tests themselves inside each project
won't change -- the Router already outputs JUnit, TO and TM output Go test
format, etc.
Does anyone object to that? Are we ok with adding `docker-compose`
infrastructure for running TO API, TO Unit, Monitor, Router, and Portal
tests outputting JUnit?
Re: [EXTERNAL] CiaB Test Compose Outputting JUnit?
Posted by "Schmidt, Andrew (Contractor)" <An...@comcast.com>.
+1 It’s a virtual standard. Lots of tools support in Jenkins.
On 1/25/19, 10:03 AM, "Gray, Jonathan" <Jo...@comcast.com> wrote:
+1 on using a standard test output format for tests
Integrating those reports into each job will have to be done by one of the privileged few who can modify the jobs since we havn't yet converted to Jenkins2 Pipeline jobs due to our dependency on the Github-PR-Builder Jenkins plugin. As such, I'd recommend we also keep around our classical text file format. Come find me on slack if you want more details on status/progress here.
Also, I'd like to see us standardize that when performing tests we use exit code != 0 to mean the tests couldn't be run/completed as opposed to test failure. With the adoption of JUnit, that will unlock a new Jenkins build status "UNSTABLE" automatically if the tests fail yet the exit codes are 0. This means that you have better visibility into test infrastructure failures versus test failures and allows you to choose if some failure is ok. I believe currently the Github-PR-Builder plugin will treat UNSTABLE the same as failure automatically.
Jonathan G
On 1/25/19, 9:41 AM, "Jeremy Mitchell" <mi...@gmail.com> wrote:
works for me
On Thu, Jan 24, 2019 at 4:13 PM Fieck, Brennan <Br...@comcast.com>
wrote:
> I'm +1. If nothing else, it doesn't add any dependencies to actually
> *running* ATC or CIAB,
> and JUnit files are technically human-readable (though I certainly
> wouldn't love doing that
> often)>
> ________________________________________
> From: Robert Butts <ro...@apache.org>
> Sent: Thursday, January 24, 2019 2:35 PM
> To: dev@trafficcontrol.apache.org
> Subject: [EXTERNAL] CiaB Test Compose Outputting JUnit?
>
> Does anyone object to making the cdn-in-a-box test compose files (e.g.
>
> https://github.com/apache/trafficcontrol/blob/master/infrastructure/cdn-in-a-box/docker-compose.traffic-ops-test.yml
> )
> output JUnit format?
>
> I made an internal fork to do that for
> `docker-compose.traffic-ops-test.yml` as well as complete
> Dockerfiles+Compose for the TM, Portal, and Router tests. At the time, I
> didn't OSS them, because I thought we shouldn't impose JUnit on TC.
>
> But in retrospect, it limits our contributions to TC for no good reason.
> Further, TC uses the Apache Jenkins server, which expects JUnit. So I'm not
> seeing a reason not to standardize all of our docker-compose test
> infrastructure around JUnit.
>
> Note I'm only talking about the `docker-compose` infrastructure to run
> tests, not the tests themselves. The tests themselves inside each project
> won't change -- the Router already outputs JUnit, TO and TM output Go test
> format, etc.
>
> Does anyone object to that? Are we ok with adding `docker-compose`
> infrastructure for running TO API, TO Unit, Monitor, Router, and Portal
> tests outputting JUnit?
>
Re: [EXTERNAL] CiaB Test Compose Outputting JUnit?
Posted by "Gray, Jonathan" <Jo...@comcast.com>.
+1 on using a standard test output format for tests
Integrating those reports into each job will have to be done by one of the privileged few who can modify the jobs since we havn't yet converted to Jenkins2 Pipeline jobs due to our dependency on the Github-PR-Builder Jenkins plugin. As such, I'd recommend we also keep around our classical text file format. Come find me on slack if you want more details on status/progress here.
Also, I'd like to see us standardize that when performing tests we use exit code != 0 to mean the tests couldn't be run/completed as opposed to test failure. With the adoption of JUnit, that will unlock a new Jenkins build status "UNSTABLE" automatically if the tests fail yet the exit codes are 0. This means that you have better visibility into test infrastructure failures versus test failures and allows you to choose if some failure is ok. I believe currently the Github-PR-Builder plugin will treat UNSTABLE the same as failure automatically.
Jonathan G
On 1/25/19, 9:41 AM, "Jeremy Mitchell" <mi...@gmail.com> wrote:
works for me
On Thu, Jan 24, 2019 at 4:13 PM Fieck, Brennan <Br...@comcast.com>
wrote:
> I'm +1. If nothing else, it doesn't add any dependencies to actually
> *running* ATC or CIAB,
> and JUnit files are technically human-readable (though I certainly
> wouldn't love doing that
> often)>
> ________________________________________
> From: Robert Butts <ro...@apache.org>
> Sent: Thursday, January 24, 2019 2:35 PM
> To: dev@trafficcontrol.apache.org
> Subject: [EXTERNAL] CiaB Test Compose Outputting JUnit?
>
> Does anyone object to making the cdn-in-a-box test compose files (e.g.
>
> https://github.com/apache/trafficcontrol/blob/master/infrastructure/cdn-in-a-box/docker-compose.traffic-ops-test.yml
> )
> output JUnit format?
>
> I made an internal fork to do that for
> `docker-compose.traffic-ops-test.yml` as well as complete
> Dockerfiles+Compose for the TM, Portal, and Router tests. At the time, I
> didn't OSS them, because I thought we shouldn't impose JUnit on TC.
>
> But in retrospect, it limits our contributions to TC for no good reason.
> Further, TC uses the Apache Jenkins server, which expects JUnit. So I'm not
> seeing a reason not to standardize all of our docker-compose test
> infrastructure around JUnit.
>
> Note I'm only talking about the `docker-compose` infrastructure to run
> tests, not the tests themselves. The tests themselves inside each project
> won't change -- the Router already outputs JUnit, TO and TM output Go test
> format, etc.
>
> Does anyone object to that? Are we ok with adding `docker-compose`
> infrastructure for running TO API, TO Unit, Monitor, Router, and Portal
> tests outputting JUnit?
>
Re: [EXTERNAL] CiaB Test Compose Outputting JUnit?
Posted by Jeremy Mitchell <mi...@gmail.com>.
works for me
On Thu, Jan 24, 2019 at 4:13 PM Fieck, Brennan <Br...@comcast.com>
wrote:
> I'm +1. If nothing else, it doesn't add any dependencies to actually
> *running* ATC or CIAB,
> and JUnit files are technically human-readable (though I certainly
> wouldn't love doing that
> often)>
> ________________________________________
> From: Robert Butts <ro...@apache.org>
> Sent: Thursday, January 24, 2019 2:35 PM
> To: dev@trafficcontrol.apache.org
> Subject: [EXTERNAL] CiaB Test Compose Outputting JUnit?
>
> Does anyone object to making the cdn-in-a-box test compose files (e.g.
>
> https://github.com/apache/trafficcontrol/blob/master/infrastructure/cdn-in-a-box/docker-compose.traffic-ops-test.yml
> )
> output JUnit format?
>
> I made an internal fork to do that for
> `docker-compose.traffic-ops-test.yml` as well as complete
> Dockerfiles+Compose for the TM, Portal, and Router tests. At the time, I
> didn't OSS them, because I thought we shouldn't impose JUnit on TC.
>
> But in retrospect, it limits our contributions to TC for no good reason.
> Further, TC uses the Apache Jenkins server, which expects JUnit. So I'm not
> seeing a reason not to standardize all of our docker-compose test
> infrastructure around JUnit.
>
> Note I'm only talking about the `docker-compose` infrastructure to run
> tests, not the tests themselves. The tests themselves inside each project
> won't change -- the Router already outputs JUnit, TO and TM output Go test
> format, etc.
>
> Does anyone object to that? Are we ok with adding `docker-compose`
> infrastructure for running TO API, TO Unit, Monitor, Router, and Portal
> tests outputting JUnit?
>