Unit Testing

[Becky Gibson <gi...@gmail.com>]

While Joe has been working on the Android CordovaWebView for testing, I
have been looking at the testing options on iOS.   Joe is using the
Selenium WebDriver for Android and there is also one for iOS.  However,
this is a WEB driver so we can only drive web UI elements. That means that
if you want to test the camera, notification or other api's that invoke the
device UI you still need a live person to interact with the device UI.
This is still easier than a completely manual test, but still not
completely automatic. There is a google code branch for a  nativeDriver
that is based on the Selenium WebDriver api.   This would allow testing
both web and native UIs.  However, it looks like the progress on this code
is only part time by the developers.

Before I start down the path of incorporating WebDriver into Cordova for
iOS I'd like to explore the options:


   1. We can create fully automated tests on iOS for almost all features
   including device specific ones such as taking a picture,
   selecting/displaying/editing a contact via iOS UI, notifications, etc.
    Obviously some things such as orientation and audio are going to require
   some intervention.   However, this means developing and maintaining
   separate tests for iOS.

   2. We can use WebDriver for iOS, Android and Blackberry.  We can only
   test the Web UI and more human intervention will be necessary but the tests
   should be fairly consistent between the platforms.  We will still need
   device specific tests for some features (such as the aformentioned iOS
   contact UI).

   3. We can investigate contributing to the native driver implementations (
   http://code.google.com/p/nativedriver/) to get more complete coverage,
   although there are currently no options for platforms other than Android
   and iOS.

   4. We can continue to use mobile-spec and enhance the manual testing
   with more tests and specific instructions.  Since Joe is putting the work
   into the Android Cordova View I assume we don't want that option at least
   for Android.


If we go the WebDriver route what language are we going to use to drive the
tests?   Java, Ruby, Python, php, perl and C# are all supported.


-becky

Re: Unit Testing

[Joe Bowser <bo...@gmail.com>]

Hey

I took a quick look at NativeDriver yesterday.  My response are inline:

On Fri, Feb 3, 2012 at 11:59 AM, Becky Gibson <gi...@gmail.com>wrote:

> While Joe has been working on the Android CordovaWebView for testing, I
> have been looking at the testing options on iOS.   Joe is using the
> Selenium WebDriver for Android and there is also one for iOS.  However,
> this is a WEB driver so we can only drive web UI elements. That means that
> if you want to test the camera, notification or other api's that invoke the
> device UI you still need a live person to interact with the device UI.
> This is still easier than a completely manual test, but still not
> completely automatic. There is a google code branch for a  nativeDriver
> that is based on the Selenium WebDriver api.   This would allow testing
> both web and native UIs.  However, it looks like the progress on this code
> is only part time by the developers.
>
> Before I start down the path of incorporating WebDriver into Cordova for
> iOS I'd like to explore the options:
>
>
>   1. We can create fully automated tests on iOS for almost all features
>   including device specific ones such as taking a picture,
>   selecting/displaying/editing a contact via iOS UI, notifications, etc.
>    Obviously some things such as orientation and audio are going to require
>   some intervention.   However, this means developing and maintaining
>   separate tests for iOS.
>
>
I would assume that this would mean separate tests for each platform.  The
platforms are too different for a unified UI testing approach.  I did take
a look at Native Driver and the thing is that it doesn't really do what I
need for unit testing/functional testing. For example, I have a test that
uses WebDriver's client only written in JUnit that grabs the "Android" text
out of a span once the WebView loads.  I want access to both the Native
code and the plugins to catch any underlying issues.  Also, Android doesn't
really have any Native UI. Unlike the other platforms, we use Android
Intents to call 3rd Party applications and we're left relying on their
cameras.  We have no control over the UI of the cameras, which can change
on the device, nor the source code.


>   2. We can use WebDriver for iOS, Android and Blackberry.  We can only
>   test the Web UI and more human intervention will be necessary but the
> tests
>   should be fairly consistent between the platforms.  We will still need
>   device specific tests for some features (such as the aformentioned iOS
>   contact UI).
>
>
This is the one that makes the most sense.  That being said, Android uses
JUnit tests and the client library and doesn't use any sort of server.
This is intentional, since I want to test both the Native parts of Android
and the Javascript interface.  If we back our tests up with a server, I
lose the ability of checking the state of native, and selenium then has no
value, since I can already test in the browser with Mobile Spec.  This is
important for core features of Cordova on Android, such as receiving
intents, or passing messages to the plugins in the PluginManager.

  3. We can investigate contributing to the native driver implementations (
>   http://code.google.com/p/nativedriver/) to get more complete coverage,
>   although there are currently no options for platforms other than Android
>   and iOS.
>
>   4. We can continue to use mobile-spec and enhance the manual testing
>   with more tests and specific instructions.  Since Joe is putting the work
>   into the Android Cordova View I assume we don't want that option at least
>   for Android.
>
>
I think we should do this for Android as well.  There are certain things
that I just can't test using my approach (i.e. Camera test still requires
user intervention to actually take the picture, Accelerometer requires us
to actually tilt the device, etc).


>
> If we go the WebDriver route what language are we going to use to drive the
> tests?   Java, Ruby, Python, php, perl and C# are all supported.
>

I use Java and I run the tests all on the client with the Android JUnit
Instrumentation.  I don't think there is a single silver bullet to testing,
sadly.  Given the fact that we are testing separate codebases for each
platform, we should have separate tests for each platform.  The goal of the
Android work that I'm doing is so that we can make the Android Development
more test driven than issue driven.  Since we write new features in Java
and do changes to the application in Java, we should write tests to check
those Java features that are independent of the JS.  That way we have a
sanity check when bugs do go around to see whether the bug is in native, or
whether it's a Javascript bug.

As far as us giving tools for users, I think we should give the users the
ability to use WebDriver in Cordova for tests, since they are supposed to
be testing on the web, and they can drive these tests in whatever language
they like.  Here's the current branch that I've been working on with
WebDriver and JUnit testing.  This may explain more what I mean when I say
that each platform should have their own tests:

https://github.com/infil00p/callback-android/tree/cordova_view

Joe