You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cordova.apache.org by ja...@apache.org on 2019/02/11 16:32:48 UTC
[cordova-contribute] branch janpio-testing created (now 1a26450)
This is an automated email from the ASF dual-hosted git repository.
janpio pushed a change to branch janpio-testing
in repository https://gitbox.apache.org/repos/asf/cordova-contribute.git.
at 1a26450 Testing
This branch includes the following new commits:
new 2fb718f empty README
new 2b0e944 short introduction
new 5f591f4 fix
new ffbb522 .editorconfig for better code and markdown auto-formatting
new 1a26450 Testing
The 5 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cordova.apache.org
For additional commands, e-mail: commits-help@cordova.apache.org
[cordova-contribute] 05/05: Testing
Posted by ja...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
janpio pushed a commit to branch janpio-testing
in repository https://gitbox.apache.org/repos/asf/cordova-contribute.git
commit 1a2645015ff81612f2c576538acb333e6679ef9e
Author: Jan Piotrowski <pi...@gmail.com>
AuthorDate: Wed Nov 28 12:17:36 2018 +0100
Testing
---
testing-notes | 100 +++++++++++++++++++++++++++
testing-plugins.md | 70 +++++++++++++++++++
testing.md | 196 +++++++++++++++++++++++++++++++++++++++++++++++++++++
3 files changed, 366 insertions(+)
diff --git a/testing-notes b/testing-notes
new file mode 100644
index 0000000..c39e736
--- /dev/null
+++ b/testing-notes
@@ -0,0 +1,100 @@
+#### https://github.com/apache/cordova-android
+
+https://github.com/apache/cordova-android/blob/ef2434188efeef868e91232188ef080acdd28e32/package.json#L21-L30
+```
+ "scripts": {
+ "test": "run-s eslint unit-tests java-unit-tests e2e-tests",
+ "eslint": "run-s -c eslint:*",
+ "eslint:scripts": "eslint bin spec test",
+ "eslint:bins": "eslint 'bin/**/*' --ignore-pattern '**/*.*' --ignore-pattern '**/gitignore' --ignore-pattern 'bin/templates/cordova/version'"
+ "unit-tests": "jasmine --config=spec/unit/jasmine.json",
+ "java-unit-tests": "node test/run_java_unit_tests.js",
+ "e2e-tests": "jasmine --config=spec/e2e/jasmine.json",
+ "cover": "istanbul cover --root bin --print detail jasmine -- --config=spec/unit/jasmine.json",
+},
+```
+AppVeyor: npm test
+Travis: - npm test
+- npm run cover
+
+
+
+
+
+#### https://github.com/apache/cordova-ios
+
+https://github.com/apache/cordova-ios/blob/62ebfbd74aff11b9fa0ca26685ee0578ecd9150f/package.json#L20-L32
+```
+ "scripts": {
+ "test": "npm run unit-tests && npm run test:component && npm run objc-tests && npm run e2e-tests",
+
+ "unit-tests": "jasmine --config=tests/spec/unit.json",
+ "test:component": "jasmine --config=tests/spec/component.json",
+ "objc-tests": "npm run objc-tests-lib && npm run objc-tests-framework",
+ "objc-tests-lib": "xcodebuild test -workspace tests/cordova-ios.xcworkspace -scheme CordovaLibTests -destination \"platform=iOS Simulator,name=iPhone 8\" CONFIGURATION_BUILD_DIR=\"`mktemp -d 2>/dev/null || mktemp -d -t 'cordova-ios'`\"",
+ "objc-tests-framework": "xcodebuild test -workspace tests/cordova-ios.xcworkspace -scheme CordovaFrameworkApp -destination \"platform=iOS Simulator,name=iPhone 8\" CONFIGURATION_BUILD_DIR=\"`mktemp -d 2>/dev/null || mktemp -d -t 'cordova-ios'`\"",
+ "e2e-tests": "jasmine tests/spec/create.spec.js",
+
+ "posttest": "npm run eslint",
+ "eslint": "eslint bin tests"
+
+ "cover": "nyc jasmine --config=tests/spec/coverage.json",
+
+ "preobjc-tests": "tests/scripts/killsim.js",
+},
+```
+AppVeyor: - npm run unit-tests
+- npm run eslint
+Travis: - npm run eslint
+ - npm run unit-tests
+ - npm run test:component
+ - npm run e2e-tests
+ - open -b com.apple.iphonesimulator
+ - npm run objc-tests
+- npm run cover
+https://github.com/apache/cordova-ios/blob/62ebfbd74aff11b9fa0ca26685ee0578ecd9150f/.travis.yml#L22-L28
+
+#### https://github.com/apache/cordova-windows
+
+https://github.com/apache/cordova-windows/blob/f687f6a58bcad5893039af991921cab4dae5bdcc/package.json#L19-L25
+```
+ "scripts": {
+ "test": "npm run eslint && npm run test-unit && npm run test-e2e",
+ "test-unit": "jasmine",
+ "test-e2e": "jasmine spec/e2e/endtoend.spec.js",
+ "cover": "istanbul cover --root bin/templates/cordova --print detail jasmine",
+ "eslint": "eslint bin && eslint template && eslint spec"
+},
+```
+AppVeyor: npm test
+Travis: - npm run eslint
+- npm run cover
+
+#### https://github.com/apache/cordova-browser
+
+https://github.com/apache/cordova-browser/blob/59690b4f9a8bd47887f61c81aa6a6aba9962dade/package.json#L19-L23
+```
+ "scripts": {
+ "eslint": "eslint bin && eslint spec",
+ "jasmine": "jasmine",
+ "test": "npm run eslint && npm run jasmine"
+},
+```
+- AppVeyor: npm test
+- Travis: npm test
+
+#### https://github.com/apache/cordova-osx
+
+https://github.com/apache/cordova-osx/blob/8a70ced07351688c75ed09c9fe77eeb9a3dc7f16/package.json#L17-L23
+
+```
+ "scripts": {
+ "test": "npm run eslint && npm run test:coverage",
+ "test:objc": "jasmine --config=tests/spec/objc.json",
+ "test:component": "jasmine --config=tests/spec/component.json",
+ "test:coverage": "nyc jasmine --config=tests/spec/coverage.json",
+ "eslint": "eslint bin tests"
+},
+```
+AppVeyor: -
+Tracis: npm test
diff --git a/testing-plugins.md b/testing-plugins.md
new file mode 100644
index 0000000..44a2c00
--- /dev/null
+++ b/testing-plugins.md
@@ -0,0 +1,70 @@
+# Testing Cordova Plugins
+
+## Writing tests
+
+### Unit Tests
+
+
+jasmine
+
+### Integration Tests
+
+1. `/tests` in app
+2. https://github.com/apache/cordova-plugin-test-framework
+3. `cdvtests/index.html`
+
+
+Install plugin and its tests in an app:
+
+```
+cordova plugin add https://github.com/apache/cordova-plugin-file-transfer.git
+cordova plugin add plugins/cordova-plugin-file-transfer/tests # local tests that were downloaded with previous commands
+cordova plugin add https://github.com/apache/cordova-plugin-test-framework
+```
+
+
+
+
+
+
+
+
+## Manual
+
+
+
+## Automated
+
+eslint
+paramedic
+ appium tests
+
+### CI
+
+Continuous Integration of Cordova plugins uses multiple providers:
+
+- Travis for Linux
+- AppVeyor for Windows
+
+There is no (or: only sometimes) direct testing on macOS (on Travis).
+
+#### Sauce Labs
+
+But integration tests are run on Saucelabs on many different devices, emulators and browsers (Android, iOS, Firefox, Safari, Chrome etc.).
+
+The tests are executed via Travis, which triggers `cordova-paramedic`.
+The tests are written in Appium.
+The list of platforms to test is configured in `.travis.yml`.
+The paramedic command is triggered after running all other tests.
+
+##### Credentials
+
+Account `snay`
+
+```yaml
+addons:
+ jwt:
+ secure: QivPLlqTVvOo3TJeHxuBOfxU6lho1I0IxQ3b68yntkEQQJko6kzleXHfgjf0a8aw8m38E3+fxaBWF1bGyucGwOLDWY8Ddt2P2xg44zdXH5EXHd9oIqAgngIdzLvUtH3Db2TbQEtIGOkrnNR2STovjqB7vHGLASQrgs4oL7r32/s=
+```
+
+encrypted SAUCE_ACCESS_KEY with https://docs.travis-ci.com/user/jwt
diff --git a/testing.md b/testing.md
new file mode 100644
index 0000000..d281b2d
--- /dev/null
+++ b/testing.md
@@ -0,0 +1,196 @@
+# Testing Cordova
+
+The aim of Cordova's developer team is to produce great and working software. To achieve this, we invested a lot of time into making Cordova testable. This document should give a general overview about our testing practices and setup.
+
+## Table of Contents
+
+- [What is the current state of the tests?](#what-is-the-current-state-of-the-tests-)
+- [What types of tests does Cordova use?](#what-types-of-tests-does-cordova-use-)
+ * [Manual Tests](#manual-tests)
+ * [Automated Tests](#automated-tests)
+ + [Syntax Checker](#syntax-checker)
+ + [Unit Tests](#unit-tests)
+ - [Code Coverage](#code-coverage)
+ + [Component Tests](#component-tests)
+ + [Integration / End to End Tests](#integration---end-to-end-tests)
+ - [Appium](#appium-interface-tests)
+- [How to implement tests?](#how-to-implement-tests-)
+ * [Tooling](#tooling)
+ * [Platforms](#platforms)
+ + [Native Unit Tests](#native-unit-tests)
+ * [Plugins](#plugins)
+ + [Plugin tests with `cordova-plugin-test-framework`](#plugin-tests-with--cordova-plugin-test-framework-)
+ + [Appium Tests](#appium-tests)
+ + [Paramedic: `cordova-paramedic`](#paramedic---cordova-paramedic-)
+ - [Sauce Labs](#sauce-labs)
+ + [Core Plugins](#core-plugins)
+ + [3rd party plugins](#3rd-party-plugins)
+- [How to run tests?](#how-to-run-tests-)
+ * [Local](#local)
+ + [Special case: Plugins](#special-case--plugins)
+ * [Continuous Integration](#continuous-integration)
+
+<!--<small><i><a href='http://ecotrust-canada.github.io/markdown-toc/'>Table of contents generated with markdown-toc</a></i></small>-->
+
+## What is the current state of the tests?
+
+If you are only interested in the [current build status](#continuous-integration) of the repositories and [how much of the code is covered by tests](#code-coverage) you can check our [Cordova Status Dashboard](https://apache.github.io/cordova-status/). It shows badges for each CI provider that indicate the current build status and a badge that contains the % of code coverage for each repository. Click each badge for more information.
+
+If you are interested in how all this works, please read on:
+
+## What types of tests does Cordova use?
+
+Cordova projects use several different types of tests:
+
+### Manual Tests
+
+- Check if some manual action produces the expected result.
+- Manual tests are often not explicitly documented, but implicitly assumed. If there is an actual implementation, it is much more "free form" as with the other test types mentioned on this page: E.g. for [plugins](#plugins), manual tests are defined as a list of buttons to tap and an "Expected result" text below the button.
+
+### Automated Tests
+
+#### Syntax Checker
+
+- Check if the code of a project follows specific syntax guidelines.
+- Usually uses [`eslint`](https://eslint.org/), which is "a fully pluggable tool for identifying and reporting on patterns in JavaScript".
+
+#### Unit Tests
+
+- Check if individual pieces of code produce the expected results.
+- All Cordova projects use [`jasmine`](https://jasmine.github.io/) to write and run **unit tests** for their JavaScript code: "Jasmine is a behavior-driven development framework for testing JavaScript code."
+- For their native code some projects use the appropriate tools of the platform to run tests: e.g. `gradle` for running Android JUnit tests, `xcodebuild test` for running [iOS XCTests](https://developer.apple.com/documentation/xctest)
+
+##### Code Coverage
+
+- Check how much of the code base is covered by the previously run Unit Tests.
+- Some Cordova projects (see the Codecov badges [here](https://github.com/apache/cordova-status)) collect code coverage by using [`nyc`](https://github.com/istanbuljs/nyc) (e.g. `cordova-android`, `cordova-android`) and report the results to [Codecov](https://codecov.io/) for tracking and visualization.
+
+#### Component Tests
+
+- Check if an individual piece of code, interacting with an external component, produces the expected results.
+- Component tests are used in a few places, and are also written with `jasmine`.
+
+#### Integration / End to End Tests
+
+- Check if multiple pieces of code or software work together as expected.
+- `jasmine` is also used to write and execute end to end tests.
+
+##### Appium Interface Tests
+
+- Check the app behavior by simulating actual user input.
+- Appium tests also use `jasmine` syntax for the general testing setup, but the [Webdriver API](https://www.w3.org/TR/webdriver1/) to simulate and drive interactions with the app on the device.
+<!-- TODO Means that the native code is being exercised. -->
+
+## How to implement tests?
+
+### Tooling
+
+TODO
+
+### Platforms
+
+TODO `/spec`
+
+#### Native Unit Tests
+
+Additionally to the usual unit tests explained above, for the platforms there can also be **native unit tests**. Those usually live in `/test(s)` and come in the form of a native test project that interacts with the main platform code. They can also be run via a `npm run ...` command that triggers a Javascript file in the test directory.
+
+For Android for example:
+- https://github.com/apache/cordova-android/blob/master/test/
+- https://github.com/apache/cordova-android/blob/master/test/run_java_unit_tests.js
+- `npm run java-unit-tests`
+
+### Plugins
+
+Plugin tests are somewhat special, as they are set up in a different way. `npm run tests` just includes the syntax check, while the real tests are run by other means:
+
+#### Native
+
+https://github.com/apache/cordova-plugin-camera/tree/master/tests/ios
+https://github.com/apache/cordova-plugin-splashscreen/tree/master/tests/ios
+https://github.com/apache/cordova-plugin-wkwebview-engine/tree/master/tests/ios
+
+#### Plugin tests with `cordova-plugin-test-framework`
+
+The actual tests can be found as `test.js` in a `/tests` folder. The automated tests, exported as `defineAutoTests`, use the familiar `jasmine` syntax, while the manual tests can be any HTML elements - usually buttons that trigger a JS function, with a description of the expected result underneath - exported via `defineManualTests`.
+
+Additionally, the `/tests` folder contains its own `package.json` and `config.xml` to create a "mini test plugin" to be used by [`cordova-plugin-test-framework`](https://github.com/apache/cordova-plugin-test-framework) to create a test interface (manual UI and `jasmine` test runner) that is viewable inside an app: Create a new app, add the plugin, add the "tests" sub-plugin and you can navigate to `cdvtests/index.html` in your app to view that test interface.
+
+#### Appium Tests
+
+A small number of plugins also implements End to End Tests with Appium. They live in `/appium-tests`.
+TODO Native code is being execised vs. JS part with all the other test types
+
+#### Paramedic: `cordova-paramedic`
+
+To automate the creating of a test app and the test execution process [`cordova-paramedic`](https://github.com/apache/cordova-paramedic) can be be used:
+
+1. Install `cordova-paramedic` (`npm install -g ...` or clone + `npm install`)
+2. Run Paramedic for the current plugin:
+ ```
+ node /tmp/paramedic/main.js
+ --config pr/$PLATFORM
+ --plugin $(pwd)
+ ```
+3. App is built, setup; Tests are run and results reported
+
+##### Sauce Labs
+
+Paramedic can also run these tests on a remote service called Sauce Labs. They offer emulators and real devices of all kind to run testing on. Both normal plugin tests and Appium tests are executed on Sauce Labs on different operating systems and operating system versions.
+
+#### Core Plugins
+
+(Almost) All Core Plugins implement tests in the previously described ways. Some focus on automated tests, some on manual ones. The plugin tests are executed on CI by the already mentioned `cordova-paramedic` on Sauce Labs.
+
+#### 3rd party plugins
+
+TODO
+
+Some have tests, not very many unfortunately.
+
+
+## How to run tests?
+
+### Local
+
+As all Cordova projects are set up as JavaScript projects, the available test commands are usually defined in the `scripts` array of `package.json`. An example from `cordova-android`:
+
+```json
+ "scripts": {
+ "test": "run-s eslint unit-tests java-unit-tests e2e-tests",
+ "eslint": "run-s -c eslint:*",
+ "eslint:scripts": "eslint bin spec test",
+ "eslint:bins": "eslint 'bin/**/*' --ignore-pattern '**/*.*' --ignore-pattern '**/gitignore' --ignore-pattern 'bin/templates/cordova/version'"
+ "unit-tests": "jasmine --config=spec/unit/jasmine.json",
+ "java-unit-tests": "node test/run_java_unit_tests.js",
+ "e2e-tests": "jasmine --config=spec/e2e/jasmine.json",
+ "cover": "istanbul cover --root bin --print detail jasmine -- --config=spec/unit/jasmine.json",
+ },
+```
+
+This defines different keywords that can be used for `npm run ...`, i.e. `npm run test`. In this case `test` is a combination of sub-commands, that can also be run individually.
+
+Contributors are expected to use those commands before, during and after making changes to the codebase to ensure that the tests still pass.
+
+#### Special case: Plugins
+
+As plugin tests are implemented in a special way, read [how to implement and run plugin tests below](#plugins).
+
+### Continuous Integration
+
+The same npm scripts are also used in CI environments to run tests. Each CI service in use usually has a configuration file in the repository under test, e.g. `.travis.yml` for Travis CI. These configuration files define the setup and commands to run on each build, which usually includes `npm run test` or similar commands to run these tests.
+
+<!--
+
+## Missing bits
+
+- Proper integration tests of CLI and platforms
+- Testing installation of 3rd party plugins
+- Testing installation of core plugins
+
+
+#### Command line
+#### Emulators or Devices
+#### Remote Emulators or Devices
+
+-->
\ No newline at end of file
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cordova.apache.org
For additional commands, e-mail: commits-help@cordova.apache.org
[cordova-contribute] 04/05: .editorconfig for better code and
markdown auto-formatting
Posted by ja...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
janpio pushed a commit to branch janpio-testing
in repository https://gitbox.apache.org/repos/asf/cordova-contribute.git
commit ffbb522051e335b1abd48a74c062491b2ff5565f
Author: Jan Piotrowski <pi...@gmail.com>
AuthorDate: Thu Aug 23 14:56:00 2018 +0200
.editorconfig for better code and markdown auto-formatting
---
.editorconfig | 17 +++++++++++++++++
README.md | 3 ++-
2 files changed, 19 insertions(+), 1 deletion(-)
diff --git a/.editorconfig b/.editorconfig
new file mode 100644
index 0000000..251c64e
--- /dev/null
+++ b/.editorconfig
@@ -0,0 +1,17 @@
+# EditorConfig helps developers define and maintain consistent coding styles between different editors and IDEs
+# editorconfig.org
+
+root = true
+
+[*]
+indent_style = space
+indent_size = 2
+
+# We recommend you to keep these unchanged
+end_of_line = lf
+charset = utf-8
+trim_trailing_whitespace = true
+insert_final_newline = true
+
+[*.md]
+trim_trailing_whitespace = false
diff --git a/README.md b/README.md
index fbe1e3c..f56ff10 100644
--- a/README.md
+++ b/README.md
@@ -4,4 +4,5 @@ This repository contains documents describing everything a contributor of Apache
> **Work in Progress**: This repository is very new and is currently very much "Work in Progress": content is created and added via Pull Requests.
-## Topics
\ No newline at end of file
+## Topics
+
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cordova.apache.org
For additional commands, e-mail: commits-help@cordova.apache.org
[cordova-contribute] 02/05: short introduction
Posted by ja...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
janpio pushed a commit to branch janpio-testing
in repository https://gitbox.apache.org/repos/asf/cordova-contribute.git
commit 2b0e944312a3850e27de16d35b2888010e0e6a71
Author: Jan Piotrowski <pi...@gmail.com>
AuthorDate: Wed Aug 22 18:25:15 2018 +0200
short introduction
---
README.md | 6 +++++-
1 file changed, 5 insertions(+), 1 deletion(-)
diff --git a/README.md b/README.md
index f0f1321..79ac7f6 100644
--- a/README.md
+++ b/README.md
@@ -1 +1,5 @@
-# cordova-contribute
\ No newline at end of file
+# cordova-contribute# cordova-contribute
+
+This repository contains documents describing everything a contributor of Apache Cordova or Community Member needs know: guidelines, policies, rules, how it's done.
+
+> **Work in Progress**: This repository is very new and is currently very much "Work in Progress": content is created and added via Pull Requests.
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cordova.apache.org
For additional commands, e-mail: commits-help@cordova.apache.org
[cordova-contribute] 03/05: fix
Posted by ja...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
janpio pushed a commit to branch janpio-testing
in repository https://gitbox.apache.org/repos/asf/cordova-contribute.git
commit 5f591f4088d22c4656efa8143e936d769c5e2f36
Author: Jan Piotrowski <pi...@gmail.com>
AuthorDate: Wed Aug 22 19:11:59 2018 +0200
fix
---
README.md | 6 ++++--
1 file changed, 4 insertions(+), 2 deletions(-)
diff --git a/README.md b/README.md
index 79ac7f6..fbe1e3c 100644
--- a/README.md
+++ b/README.md
@@ -1,5 +1,7 @@
-# cordova-contribute# cordova-contribute
+# cordova-contribute
-This repository contains documents describing everything a contributor of Apache Cordova or Community Member needs know: guidelines, policies, rules, how it's done.
+This repository contains documents describing everything a contributor of Apache Cordova or Community Member needs to know: guidelines, policies, rules, how it's done.
> **Work in Progress**: This repository is very new and is currently very much "Work in Progress": content is created and added via Pull Requests.
+
+## Topics
\ No newline at end of file
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cordova.apache.org
For additional commands, e-mail: commits-help@cordova.apache.org
[cordova-contribute] 01/05: empty README
Posted by ja...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
janpio pushed a commit to branch janpio-testing
in repository https://gitbox.apache.org/repos/asf/cordova-contribute.git
commit 2fb718ffa8e5366639ca37349d0dfaf173d76c85
Author: Jan Piotrowski <pi...@gmail.com>
AuthorDate: Wed Aug 22 15:04:03 2018 +0200
empty README
---
README.md | 1 +
1 file changed, 1 insertion(+)
diff --git a/README.md b/README.md
new file mode 100644
index 0000000..f0f1321
--- /dev/null
+++ b/README.md
@@ -0,0 +1 @@
+# cordova-contribute
\ No newline at end of file
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cordova.apache.org
For additional commands, e-mail: commits-help@cordova.apache.org