Click here to Skip to main content
Click here to Skip to main content

Tagged as

Go to top

Automated Android* Application Testing

, 22 Jul 2014
In this article I will review several tools for automated functional testing. I'll review only tools included in Android SDK or distributed under Open Source license.

Editorial Note

This article is in the Product Showcase section for our sponsors at CodeProject. These reviews are intended to provide you with information on products and services that we consider useful and of value to developers.

Introduction

Testing is an important part of the application development process. For Android, it is particularly important because the devices are very different from each other in the following ways:

  • Screen size and resolution
  • Android version
  • Form factor
  • Instruction set of the processor
  • The presence of the front camera, NFC, external keyboard, etc.

Your Android application should be tested on many devices.

Application testing processes include many types of testing. Let's take a look at manual functional testing. The tester needs to carefully check all the functionality and reset the device to an initial state. The tester repeats these steps for each application and each device. Done manually, this process is very time-consuming.

Automated functional testing can be performed regularly without additional costs. For example, test a build every night on all devices, analyze the results in the morning, and fix the bugs.

In this article I will review several tools for automated functional testing. I'll review only tools included in Android SDK or distributed under Open Source license.

The Concept of Automated Testing

Our goal is to automate actions that are performed manually for maximum precision. Let's review these actions. We'll use several applications and several Android devices.

For each application and each device we need to perform the following steps:

  1. Install the application on the device
  2. Launch the application
  3. Test the application using the selected method
  4. Remove the application
  5. Reset the device to the initial stage

At each step you need to collect and analyze data such as logs and screenshots. Below, we’ll discuss the tools that automate these steps.

Control Android devices

First, you need to select the computer that will run the automated test and install the Android SDK on it. I'll use a desktop computer with Linux* as an example. You need to disable lock screen and increase "time before sleep mode" to the maximum level on each device.For some testing methods, you need to disable screen orientation change.

There are two utilities in the Android SDK to control Android devices: ADB and monkeyrunner*. I'll describe in detail how to automate actions that are used in manual testing.

Control the Android device using ADB

ADB (Android Debug Bridge) is a command line tool to control Android devices. The ADB home page is: http://developer.android.com/tools/help/adb.html

The ADB tool is located in the directory <android_sdk>/platform-tools/.You should put this directory in the PATH environment variable.

Checking the ADB installation

Install and configure the Android SDK, then connect Android devices to your computer and run the command:

adb devices

This command will list all plugged-in Android devices. If the list of devices is not empty, then ADB is working properly.

Working with multiple devices

You should use "-s" parameter to specify which device ADB should use.

adb -s [serial_number] [command]

For example:

adb -s [serial_number] logcat

The serial number of the device can be obtained from the output of «adb devices» command. Parameter -s allows you to work simultaneously with multiple connected devices.

Basic ADB commands

Open a console on the device:

adb shell

Run a command on the device:

adb shell [command]

Many standard Linux utilities are included in Android: ls, cat, dmesg, ...

Install the application from the apk file:

adb install example.apk

Uninstall the application:

adb uninstall [package]

Get a package name from the apk file:

aapt dump badging example.apk | grep "Package"

Download a file from the device to the computer:

adb pull [path-on-device] [file]

Upload a file from the computer to the device:

adb push [file] [path-on-device]

Note:
Only read access is allowed to most directories on the Android device. Write access is allowed to /sdcard (but you can’t run programs from this directory) and /data/local/tmp.

Start an application:

adb shell am start -n [package]/[activity]

Run the specified activity.

You can extract the activity name from the apk file:

aapt dump badging example.apk | grep "launchable-activity"

Reading logs

Logcat is a command that reads logs from Android devices.

Logcat homepage:http://developer.android.com/tools/help/logcat.html

Read logs from the device (blocks until you press Ctrl-C):

adb logcat

Clear log buffer on the device:

adb logcat -c

Dump log buffer on the device (shows current buffer contents, non-blocking):

adb logcat -d

Example:

adb logcat -c # clear the buffer log
# Action
adb logcat -d > file.log # save the current contents of the log buffer to file.log

Capture screenshot using screencap

The screencap utility saves the current contents of the screen to a graphic file:

adb shell screencap /sdcard/screen.png
adb pull /sdcard/screen.png screen.png
adb shell rm /sdcard /screen.png

The screencap utility is available on phones with Android 4.x and above. On previous Android versions you can capture screenshots using monkeyrunner.

BASH script to test the application using ADB

Script: app_test.sh

Control the Android device with MonkeyRunner

The monkeyrunner tool provides APIs for scripts that control Android devices. You can write a Python* script with monkeyrunner that installs an Android application, launches it, emulates user actions, takes screenshots, and saves them to your computer. Monkeyrunner uses Jython* to run scripts.

The monkeyrunner home page and API reference: http://developer.android.com/tools/help/monkeyrunner_concepts.html

Reading logs with monkeyrunner

File log.py:

# coding: utf-8
from com.android.monkeyrunner import MonkeyRunner, MonkeyDevice

def log(fn, device):
    msg = device.shell('logcat -d')
    f_log = open(fn, 'at')
    if msg is None:
        msg = 'None'
    f_log.write(msg.encode('utf-8'))
    f_log.close()    
    device.shell('logcat -c')

if __name__ == '__main__':
    device = MonkeyRunner.waitForConnection()
    device.shell('logcat -c') # Clear logs buffer
    # ...
    log('example.log', device) # Write logs

Starting:

monkeyrunner log.py

The script will write logs to a file example.log in the current directory.

Capture screenshot using MonkeyRunner

File screen.py:

# coding: utf-8
from com.android.monkeyrunner import MonkeyRunner, MonkeyDevice

if __name__ == '__main__':
    device = MonkeyRunner.waitForConnection()
    image = device.takeSnapshot()
    image.writeToFile('screenshot.png','png')

Starting:

monkeyrunner screen.py

The script takes a screenshot and saves it to a filescreenshot.png in the current directory.

Example of device control using monkeyrunner

Script: monkeyrunner_test.py

Starting:

monkeyrunner monkeyrunner_test.py

Automated Testing Methods

Testing with Monkey*

Imagine that the device you are testing is given to a very active and creative monkey—the Monkey tool is designed to simulate this situation.The Monkey tool, a part of the Android SDK, sends a stream of random user events.Command line parameters specify the number of user actions, the ratio of each event type, and a name of a package (so Monkey will not go beyond the limits of the tested application and will not start sending SMS to all the contacts in the Address Book).

Examples and a list of parameters are on the Monkey home page: http://developer.android.com/tools/help/monkey.html

The main advantage of the Monkey tool is zero maintenance costs. Furthermore, stress testing can detect non-trivial bugs.

Disadvantages of testing with the Monkey tool:

  • Monkey can’t simulate complex workloads such as authentication. In such cases, application functionality remains untested.
  • Games with complex control that require quick reactions and complex gestures will be completed at the beginning, or do not begin.
  • It’s very difficult to reproduce errors found by Monkey.
  • Monkey doesn’t check application status during the test.

Automated testing with Monkey is a good starting point for any application. It is possible that this method will show adequate results for a particular application.If the testing quality is low, you should use other testing methods.

Testing with MonkeyRunner

You can not only develop Android device control scripts using MonkeyRunner, you can also write scripts to test the application on a specific device.

Advantages

  • Flexible.

Disadvantages:

  • The complexity of writing scripts, even in simple cases.

Developing monkeyrunner scripts takes a lot of time, so this method is usually not justified.However, in special cases, this method might work.

Testing with getevent and sendevent

Getevent and sendevent utilities allow the user to record a sequence of events and replay the sequence.Root permissions are not required to run these tools.

Advantages

  • Event sequences can be recorded without additional costs in manual testing, if it is carried out.
  • Programming skills are not required to record an event sequence.

Disadvantages:

  • Sequence should be recorded separately for each application and each device.If you change the interface of an application, all recorded actions should be redone.
  • This method doesn’t check application status during the test. If application response is delayed (for example, web page loading), the testing results will be incorrect.
  • Rapid and complex sequences will be played longer than recorded. So it is not always suitable for testing dynamic games where response time is critical.

Record the sequence of events:

# Record event sequence
# Do actions on the device, press Ctrl-C to finish
adb shell getevent -t > events.txt
# Convert event sequence to script
./decode_events.py events.txt > events.sh
# Load the script to the device
adb push events.sh /data/local/tmp/
# Set the permissions
adb shell chmod 755 /data/local/tmp/events.sh
# Run script
adb shell sh /data/local/tmp/events.sh

Script: decode_events.py

Replay the recorded sequence of events on the device.

Testing with Robotium*

Robotium is not a part of Android SDK, and it’s distributed under the Open Source license. The Robotium home page is: http://code.google.com/p/robotium/.

Robotium scripts define actions on the application UI level rather than the input device level.

For example, a script needs to tap on an «OK» button. The monkeyrunner script will be implemented as "Tap at the screen point (x0, y0)". A Robotium script will be implemented as "Press the button with the text "OK" ."

When actions are described at the interface level, the testing script can be made independent of interface layout, screen resolution, and orientation.

Additionally, Robotium allows you to check the application response to the action.For example, after clicking the «OK» button, the list with the "Item 1" item should appear.You can check list element names with Robotium. If you check the application state after each step, it is easy to find at which step the error occurred.

Disadvantages

  • You need to develop a test script in Java* for each application. This requires programming skills and time.
  • When the application interface changes, the event sequence must be redone.
  • It’s harder to write Robotium scripts compared to recording events using getevent / sendevent.

In general, Robotium allows you to develop the highest quality test cases with adequate cost.

Testing methods comparison

Testing method Advantages Disadvantages

Monkey - a stream of random user actions

No maintenance costs.
Independent of device.
Stress testing can detect non-trivial errors.

Quality of testing varies from application to application.
Bug reports are hard to reproduce.

Monkey doesn’t check the state of the application during testing.

monkeyrunner – device control script

Flexibility

The complexity of scripting, even for simple applications.

getevent/sendevent - record/replay user actions

Programming skills are not required to record event sequence.

Recorded sequence of actions will only fit one device at a fixed screen orientation.

When the application interface changes, the event sequence must be redone.

This method doesn’t check state of application during testing.

Robotium - test script API to verify the status

Actions are described at the application UI level.

Script may be independent of the screen resolution and screen orientation.

Script can check application state after an action.

The complexity of writing scripts in Java.

If you change the application interface, you will need to modify the script.

Result analysis

Now we need to analyze the logs and screenshots collected during the automated testing process for errors.

Log analysis

You can do a search for strings:

  • I/DEBUG
  • FATAL EXCEPTION
  • WIN DEATH

This list can be extended with error messages that were found during manual testing.

Screenshot analysis

You can prepare a series of screenshots at key moments of testing and compare them to the screen content during automated testing. This will determine if the automated testing process is running correctly.

It's useful to compare the initial screenshot with the screenshot after the application launches. This helps to detect cases when the application silently fails.

Monkeyrunner allows you to compare two screenshots with the specified tolerance in percents:

image1 = device.takeSnapshot()
# ...
image2 = device.takeSnapshot()
if image2.sameAs(image1, 0.1):
    print 'image1 and image2 are the same (10%)'

Unfortunately, there is no MonkeyImage API to load an image from a file. You can write a custom function to compare images using, for example, the Python* Imaging Library.

Reset the device to the initial state

Devices should be returned to the initial state after testing.This can be achieved in several ways:

  • Press the «Back» button multiple times.
  • Reboot the device.
  • Restart the zygote process.

The first option is usually the most appropriate.

Press the Back button multiple times

Press the "Back" button using monkeyrunner:

for i in xrange(0, 10):
    device.press('KEYCODE_BACK', MonkeyDevice.DOWN_AND_UP)
    time.sleep(0.5)

In practice, this is a good option because it emulates the behavior of a real user.

Conclusion

In this article, we covered some methods of automated testing for Android applications. We reviewed advantages and disadvantages of automated testing methods.

We also discussed the Monkey and monkeyrunner tools included in the Android SDK and the Robotium tool.

Automated testing does not replace other types of testing.A properly constructed testing process that combines different testing methods including automated testing is an essential part of a high quality application development process.

Other Related Articles and Resources

License

This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Share

About the Author

Android on Intel

United States United States
No Biography provided

Comments and Discussions

| Advertise | Privacy | Mobile
Web02 | 2.8.140916.1 | Last Updated 22 Jul 2014
Article Copyright 2014 by Android on Intel
Everything else Copyright © CodeProject, 1999-2014
Terms of Service
Layout: fixed | fluid