Structured automated testing of mobile apps is still not a part of every project.  A majority of the tools that are available are either completely iOS or Android; not both.

When Xamarin released Test Cloud, a cloud-based platform for testing apps on a wide range of Android and iOS devices, we thought we could give it a try and see whether it would be a good fit for us. Headspring is a .NET and C# shop, and Xamarin Test Cloud would allow us to write tests in C#.

Because of our knowledge in .NET, we wanted to try the new Xamarin.UITest framework to write tests with C# and NUnit. To make things more complicated, we decided to test an Ionic / Cordova hybrid app instead of a fully native or Xamarin C# app.

Xamarin.UITest

To get started with Xamarin.UITest you will have to set up a NUnit test project. You can do it either in Visual Studio for Android or in Xamarin studio which supports both iOS and Android. We used Xamarin Studio for Android.

To run tests, you will need to download the Xamarin Android Player which is really fast compared to Google’s stock emulators. (Download it here.)

To start, set up your NUnit test project and write the necessary boilerplate code to get your Android APK file deployed to the emulator.

1. Open Xamarin Studio and select “New Solution”.

2. Select “NUnit Project” and name your project.

3. Next, add the NUnit and Xamarin.UITest packages by right clicking on the “Packages” folder in the project.

4. Add your .apk file to the project folder. Next, drag and drop your .apk file into the solution. Be sure to specify your Test Cloud API key as well.

5. Set the location of the Android package file and your Test Cloud API key in your test fixtures setup method.

Now that you’ve created the NUnit test project, now set up a simple test method using REPL. (You can use the very useful REPL to identify the UI elements for your app.)

Run the test. Your app will be launched in the emulator and a terminal will open up. In the terminal window, type “app.Query(c => c.All())”, to get an output of all of the elements on the current screen.

The output looks like this:

Full log file: /var/folders/c4/wvkkcw8j0c5_lz6_3ctd92640000gn/T/uitest/log-2015-02-11_08-38-49-303.txt
Android test running Xamarin.UITest version: 5
Initializing Android app on device 10.71.34.101:5555.
Running activated version with api key from ConfigureApp

 

App has been initialized to the 'app' variable.
Exit REPL with ctrl-c or see help for more commands.

>>>
>>> app.Query(c=>c.All())
Query for All() gave 6 results.
[
{
"Id": null,
"Description": "com.android.internal.policy.impl.PhoneWindow$DecorView{52977ef4 V.E..... R....... 0,0-768,1184}",
"Rect": {
"Width": 768.0,
"Height": 1184.0,
"X": 0.0,
"Y": 0.0,
"CenterX": 384.0,
"CenterY": 592.0
},
"Label": null,
"Text": null,
"Class": "com.android.internal.policy.impl.PhoneWindow$DecorView",
"Enabled": true
},
{
"Id": null,
"Description": "android.widget.LinearLayout{52870938 V.E..... ........ 0,0-768,1184}",
"Rect": {
"Width": 768.0,
"Height": 1184.0,
"X": 0.0,
"Y": 0.0,
"CenterX": 384.0,
"CenterY": 592.0
},
"Label": null,
"Text": null,
"Class": "android.widget.LinearLayout",
"Enabled": true
},
{
"Id": "action_mode_bar_stub",
"Description": "android.view.ViewStub{5287d8a0 G.E..... ......I. 0,0-0,0 #102030e}",
"Rect": {
"Width": 0.0,
"Height": 0.0,
"X": 0.0,
"Y": 0.0,
"CenterX": 0.0,
"CenterY": 0.0
},

 

How to Work with Hybrid Apps

You’ve probably noticed something peculiar about the output from the REPL command. Since we’re developing a hybrid app, all the UI elements are rendered inside a WebView. To see the actual view elements you have to look inside the WebView. This is possible through Chrome for Android, which we can use to visually identify the elements inside the DOM. (Refer to the link to find out how to do this.)

Once you have found the UI element you want to use in your tests, you have to formulate a CSS selector to uniquely identify the element you want to call that works on both Chrome and Safari. After that, you can use Xamarin’s easy-to-use API to interact with our app. For example, a call to tap an element looks like this:

app.Tap (c => c.WebView().Css("input[type=username]"));

 

Entering text into the input field looks like:

app.EnterText("sample@example.com");

 

Here is a sample test that enters a username and password into two fields on a screen:

[Test()]
public void AvalidLogin()
{
app.WaitForElement (c=>c.WebView().Css("input.ng-pristine.ng-valid"), timeout: TimeSpan.FromSeconds(15));
app.Tap (c => c.WebView ().Css ("input[type=username]"));// Tap Email
app.EnterText(username); // Enter email id
app.WaitForElement (c=>c.WebView().Css("input[type=password]"), timeout: TimeSpan.FromSeconds(15));
app.Tap( c => c.WebView().Css ("input[type=password]"));// Tap password  app.EnterText(password);//Enter password
app.Tap (c => c.WebView().Css("button[Id=Submit]" ));// Tap submit
}

 

Notice the calls to “app.WaitforElement()” in the test method. Xamarin’s test APIs provide a very helpful “WaitForElement()” method to search for the element, and if it doesn’t find the element it will throw a timeout exception. Though the test framework will typically find your element before the timeout, you will want your timeout to be long enough to account for delays on slower devices. In this example, I chose 15 seconds. You will want to choose a number that works best for you. In our case, if an element doesn’t appear on the screen within the very generous 15 second timeout, then it will probably never show up, and we have a problem.

In addition, it would be helpful to insert a number of named screenshots in to each test case so that you can see how the app screen looks on the device at any given point during the test. If a test breaks, you will know where it broke by looking at the screen shots. You can use the “Screenshot()” method to do this:

app.Screenshot(“Login Screen”);
app.Screenshot(“Entered username”);

 

Uploading to Test Cloud

Before we upload to the Test Cloud make sure the test is running in the emulator and, if possible, on the actual device. Running on the emulator locally is a must. It creates a binary file which you will upload to the Test Cloud. Keep in mind that you should run the tests on the emulator when you add any new tests or change something in the old tests before uploading to the Test Cloud.

The actual fun starts when you upload your app and corresponding test script to Xamarin Test Cloud and give it a run on multiple devices.

In the Test Cloud we can define a set of devices to deploy to and run the test on all of them.

Below is the video how tests look in Xamarin Test Cloud.

 

 

 

Upsides and Downsides

Xamarin.UI Test is a powerful tool for UI testing of any kind of mobile app. With the Xamarin Platform and Test Cloud, you have the tools to provide your clients with a quality app delivered across multiple Android and iOS devices, different form factors, and OS versions. Test Cloud doesn’t support the Windows Phone yet, but hopefully that will come soon.

The main downside is the price tag for Xamarin Test Cloud (see Xamarin’s website for more details). Also, only one set of tests can be run at the same time in the cloud, which means tests will be queued up, blocking CI builds.

But all-in-all, the experience has been positive. Besides some initial hiccups getting started, which the Xamarin support team helped me through; writing and running tests has been pretty painless. It has been an advantage to our team to have this resource.

Let's Talk