[GitHub] chetanmeh commented on issue #3572: Run full set of tests using Travis stages/jobs

[GitBox <gi...@apache.org>]

chetanmeh commented on issue #3572: Run full set of tests using Travis stages/jobs
URL: https://github.com/apache/incubator-openwhisk/issues/3572#issuecomment-386519992
 
 
   I did some analysis of the time taken in a typical travis build. Detailed break up can be seen [here][1]
   
   For purpose of this break up a custom [branch][2] was created where the `build.sh` was broken into sub scripts 
   
   1. scan - scalaFmt, scanCode
   2. couch-setup
   3. compile - common and core
   4. distDocker
   5. runDocker
   6. runTests
   7. Collect logs
   
   This was done as travis report explicit time per script entry in `script` block. Based on reports from [gradle][3] following summary can be made
   
   ### Travis Time Breakup
   
   | Phase | Time (mins)|
   |-------|-------------|
   | Base| 12|
   | Docker work| 6|
   | Code scan| 2|
   | Test db only|11|
   | Test require docker|12|
   
   Further breakup
   * Base ~12 mins
       * Install 62 sec
       * Couch 54 sec
       * Compile Core 145 sec (2.4 mins)
       * Compile Tests 386 sec (6.5 mins)
       * logs 50 sec
   * Docker ~ 6 mins
       * Dist Docker 170 secs
       * Run all docker 180 secs
   * Tests ~ 33 mins
       * Test just needing DB - 670 sec (11.15 min)
       * Test needing full docker setup (excluding previous) - 733 sec (12.2 min)
   
   ### Segregating Tests
   
   In genral convention so far was to have tests under `system` package to depend on whole setup and tests outside just relying on database. However as per [reports][1] it can be seen that some tests in following package do depend on other parts
   
   * whisk.core.controller.test
       * ControllerApiTests should ensure controller returns info
   * whisk.core.database.test
       * CacheConcurrencyTests
   * whisk.core.admin
       * WskAdminTests Wsk Admin CLI should verify guest account installed correctly
   
   Probably we can move them to system package
   
   Packages requiring complete setup
   
   * apigw.healthtests
   * services
   * system.basic
   * system.rest
   * whisk.core.limits
   * whisk.core.cli.test
   * whisk.core.apigw.actions.test
   
   [1]: https://docs.google.com/spreadsheets/d/e/2PACX-1vSujt396CMPJhvnsuDA1pHvITKrXFMzD5i1OqYwqXAxa4aif1gWakXib0rKnST351FhPqmo6VTgShWI/pubhtml#
   [2]: https://github.com/apache/incubator-openwhisk/compare/master...chetanmeh:test-db-only
   [3]: https://scans.gradle.com/s/o2fnhjckyaajw/performance/execution

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services