My project

For over a year I have been working on a project that includes building a website to publish a custom type of documents and automate the processes around these documents.

Regression testing

One of the chores to be performed by a functional analyst is to guard the quality of the end product while it is being developed. To guarantee that everything that works properly does not brake, every functional test ideally has to be re-executed every time some changes are committed - which is a boring job if done by hand.

One solution for this problem is continuous integration with continuous testing: a build server takes care of executing the whole collection of tests ideally after every commit, or at least every day. The collection of tests comprises: build tests, unit tests, regression tests.

Hudson - Jenkins

Hudson is a much used tool that can execute these daily builds on a server. See http://hudson-ci.org. The tool is able to run on Windows, Linux, etc. and is free / open source (MIT license). Hudson is a nice tool to use and not too complex.
Jenkins is an equivalent split-off free of trademarks owned by Oracle.

Goal

In my project, I wanted to execute all functional regression tests in a regular and automatic way by a Hudson server.

Manual Selenium tests

The project I am working on makes use of a complex ACL (Access Control List) for the presence of menu items per users' role: So, depending on the role(s) of the current user he gets to see a different list of menu items.
The following is an easy way to test if the ACL and menu implementation is correct: log in as a user which only has one role, and check if the correct menus appear. Then repeat this for every role.
In my project we had about 24 menu items and 12 roles, so this meant in total 288 tests. Clearly, this is a case where you have the choice between automating the tests, or skipping them ...

The Selenium IDE

The first step towards the automatisation of this test is the use of the Selenium IDE plug-in for Firefox.

Activate this Firefox plug-in and a window appears in which you can interactively develop and run a test program. The test program consists of steps written in a kind of programming language: Selenese. It is possible to execute the program in one go, or single step through. It is also possible to set breakpoints. The Selenese language does not include instructions for loops and conditions.

Testprograms are saved on disk as a HTML file where the steps are contained in a table with 3 columns and hence are editable in a text editor. This readable textual storage format is practical since it makes Subversions diff function result readable.

The Selenium IDE can organise different test programs in suites, which can be run by a single command.

The user interface of the Selenium IDE is quite simple to grasp and it supports enough functionality, but shows a few usability problems that need to get used to.

The Selenium IDE has a record button which allows one to register all manual actions on the browser, just like a macro recorder. This is a very practical way to quickly add some steps to your program, but it is surely not a suitable way to record a whole test; manual entry (or editing) of program steps remains necessary.

A test in Selenese

Here follows an example of a test program:

open	/login
type	username	expertA
type	password	expertA
clickAndWait	//input[@value='Log in']
assertTextNotPresent	Authentication failed
clickAndWait	css=a.sitelink[href*=en]
verifyTextPresent	Startpage
verifyTextPresent	Workbench
verifyTextPresent	Profiles
verifyTextPresent	Search
verifyTextPresent	List of sectors
verifyTextPresent	List of profiles
verifyTextNotPresent	User management
verifyTextNotPresent	Translations

The first column is the command and next two columns are the parameters.

Detailed discussion of the test in Selenese

Let's go over the steps in above program one by one:

open

/login

The "open" command opens a URL, in this case a relative address. The base address of the website can be entered in the Selenium IDE. In my project this is http://localhost:8888, so in this case Selenium is opening http://localhost:8888/login.

type	username	expertA
type	password	expertA

In the next two steps, Selenium types the user name and password in their input boxes. The right boxes are found by a so-called locator.
The HTML page under test contains the following input element:

<input type="text" name="username" id="username" style="width: 20em;"/>

There are many ways to write a locator in Selenese, but the most simple way is to refer to the id of a HTML element.

clickAndWait

//input[@value='Log in']

This step simulates a click on a button. The HTML page contains this input element:

<input type="submit" class="dsyfrm-primaryaction" value="Log in"/>

The locator is a XPATH statement, identifiable by the first two characters "//". Locators written in XPATH are the most versatile, but are quite hard to write and read for the novice. In this case the meaning of the XPATH statement is: locate an input element somewhere on the page that conforms the condition that it has an attribute with "value" equals "Log in".

The "AndWait" (behind the "click") makes Selenium wait until Firefox has loaded the complete next page after the button was clicked, before going on with the next step of the test program. There is also a simple "click" command, which is useful for e.g. marking a checkbox.

assertTextNotPresent

Authentication failed

An assert statement tests a condition, in this case the absence of given text on the page. An assert statement that fails causes the remainder of the test not to be executed any more. Selenium also knows a similar verify statement, which reports the failure, but does not interrupt the test program.

In this case we used the assert statement, since the remainder of the test is useless when not logged in.

clickAndWait

css=a.sitelink[href*=nl]

An other type of locator is based on CSS. In this case the page contains the following HTML:

<a class="sitelink" href="competent-nl/index.html">Competent</a>

Hence the locator refers to an "a" element with the attribute class "sitelink". The condition between square brackets follows the new CSS3 syntax.

verifyTextPresent	List of sectors
verifyTextPresent	List of profiles
verifyTextNotPresent	User management
verifyTextNotPresent	Translations

These steps test the presence of absence of menu items - which is the goal of the test in the first place. Remark that these are verify statements, not asserts. This because the result of this test should be an overview of all menu item problems, not just the first one.

Selenium tests are brittle

The Firefox plug-in Firebug is indispensible when creating a Selenium test. Firebug allows inspecting the structure of the current HTML page, inclusive the active CSS, making it easy to write e.g. an Xpath expression or CSS locator.

The original stakeholder requirements of the project do not contain any IDs for the input elements for the loginname and the password; and still we use these values in the test. So, a small change in the ID of the log in fields would fail the test completely, while the original stakeholder requirements are still fulfilled. In other words, not only the requirements are tested, but also a certain implementation.

Hence, this test is brittle, which is one of the most important problems of using Selenium for testing. Also, this makes clear that Selenium is suited better for regression tests than for functional testing.

Covering all functionality in a test written in Selenese is simple, but you have to optimize continuously to make sure that the test will not fail when some small detail changes in the HTML coding. Making the test independent of implementation choices is especially difficult when the HTML is generated by a CMS - if a CMS generates an ID like "field_27", then you know this is not a stable ID and you should not use it in the test.

Much time will be spend on making Selenium tests robust. The technique to do so requires experience, there is a learning process involved.

Automatic Selenium tests

Running tests (or test-suites) with the Selenium IDE is a long-winding manual operation. There is also another way, i.e. by converting the tests to a (real) programming language.

Perform as follows:
Start by developing a testing program in Selenese with the IDE, as described above. Once the test works, you can convert it to one of the following languages by activating a menu item in the IDE:

Java+JUnit

Java+TestNG

Groovy

Perl

PHP

Python

Ruby

My experience limits itself to Java + JUnit, so that is what I describe here.

Converting a test to Java + JUnit delivers Java source code for a class which is a JUnit test. Load the class in your favorite Java IDE and run it as a JUnit test; in Eclipse you only have to right-clickand select Run As -> JUnit test in the pop-up menu. The test will only run if you first start the so-called Selenium server: this means that you have to download a jar and run it.

The advantages of using Java for Selenium

Suppose that the name of a test object will appear on a webpage after some asynchronous communication (e.g. like the arrival of an email in the inbox). This is a practical example of a text that does not appear on a page by itself, but only after a page refresh.

The Selenium IDE does not support this situation, since it can not test conditions. The only solution would be to build in a pause that is surely long enough for the text to arraive. This leads to the following Selenese test:

open	nl/workbench.html
pause	12000
refreshAndWait
assertPresent	test object 258
clickAndWait	test object 258

The length of the pause is given in milli-seconds.

We get more control over the flow of the test by using Java, because we can split the pause up and check if the sought after text already appeared:

selenium.open("nl/workbench.html");

for (int second = 0;; second+=2) {

    if (second >= 60) fail("timeout");

    try {

        if (selenium.isTextPresent("test object 258")) break;

    } catch (Exception e) {}

    Thread.sleep(2000);

    selenium.refresh();

    selenium.waitForPageToLoad("30000");

}

selenium.click("test object 258");

selenium.waitForPageToLoad("30000");

The page is refreshed every 2 seconds within the for loop, until the text appears. Additionally, there is a timeout of one minuut - if the text does not appear at all, then the test will fail anyhow.

By using a for loop in the way descibed above, the test will last much shorter (than the worst-case 12s of the version in Selenese) and it still works correctly if there would be exceptionally large delays due to unrelated background processes.

Running Java JUnit tests in Hudson

An other advantage of the conversion of Selenese tests to Java is that this opens the possibility to run them automatically by the Hudson Continuous Integration tool. In many projects Hudson is used to test the build and to run unit tests.
It is a small effort to ask the Hudson manager to run the Selenium tests too.

The daily run of my first tests in Hudson delivered the following graph. The failed tests are red, the total number of tests is depicted as the top of the blue area.

In the beginning, there was no stability, and even after 48 days some tests were failing.

Other browsers

The Selenium IDE only exists for Firefox, but if you transfer the tests to Java (or any other supported language) then it is as easy to run the tests with another browser.

It must be fairly simple to configure Hudson to run all tests with Firefox as well as IExplorer, but I did not do that yet.

The disadvantages of using Java for Selenium

In the Selenium IDE you can try out every step interactively, adapt and try again. In Java that is not possible - you only have the possibilities of the Java debugger.

This makes it difficult to create a complex test starting from an empty page. Hence, it is easier to get the test working in the Selenium IDE and only then convert it to Java. This requires some optimalisations, as described before, e.g. the way to convert long pauses to loops with conditions.

Debugging of Java Selenium tests

Eclipse has the possibility to run every program with the debugger, hence also unit tests. You can set breakpoints, single step through the code, inspect variables, etc. A computer with two screens is very practical to monitor the web application at the same time as the Eclipse debugger.

The Java debugger can single step through the Selenium test and you can inspect the result after each step. But you can not interfere with the sequence of the steps in the test: you can not execute a step twice, or change a step and execute the changed step again. These are possible in the Selenium IDE however.

Supported techniques

It is entirely possible to use Selenium to test a website build with a CMS. The XPath locators are very flexible and allow localising any element on an automatically generated HTML page, even if the ID attributes are not suitable for use.

Selenium also includes the means to test websites that use AJAX. Just like you can use "ClickAndWait" to simulate clicking a link that opens a complete new page, you can use "Click" to activate an element that causes an AJAX refresh. In the latter case, use "WaitForText" or "WaitForElement" to wait for the text or element that should appear on the already loaded page.
Of course, the downside is that you have to know whether AJAX is used for every case to be able to write the test. In other words: the test is implementation dependent.

Also the well-known JavaScript libraries like JQuery do not pose any problem, but embedded Flash / Flex or Java applets are not controllable.

My last project contains one complex screen written with Flex and I did not yet succeed to test this with Selenium. Some research on the internet teached me that this would only be possible by building an interface in the Flex application.

Test data

Consider the example case where you can create a special kind of document at a website which then has to go through a multi-stage validation process in cooperation with multiple persons in multiple roles.
If you want to test this functionality, then you have to start by having Selenium log in as a "creator" and create the document. Then you have Selenium log in as "reviewer" with the intention to review the document's contents. So, the reviewer has to be able to locate the document. Hence the document needs a name to be able to find it.
You could simply name the document "test doc". But after a while you will then have multiple documents with the same name, e.g. caused by an interrupted test. Hence, it would be better to make up a different name at every test run.

It is possible to run Selenium tests on a cloud of computers, in which case the test runs on multiple computers at the same time, e.g. one computer runs the tests in Firefox and another in IExplorer.

Conclusion: The ideal test is independent of the state of any data. And an ideal test is re-entrant.

A simple technique is to use a pseudo random number generator to generate a test document name and store it in a variable.
For example in Selenese we can use the Javascript function to obtain the current date and time. The milliseconds of this form a pseudo random number:

storeEval	var d=new Date(); d.getMilliseconds();	myRandom
storeEval	"test document ${myRandom}"	docName

The variable "docName" contains the name. We can use this name e.g. to type it in an input box as follows:

type

name

${docName}

In Java we can make use of the built-in random number generator:

Random rn = new Random();

int number = rn.nextInt(8999) + 1000;

String docName = "test document " + number;

Screen shots

When tests fail, often it is not very clear why they fail because the error message does not give any information about what happened before. This is especially a problem with tests executed by a Hudson server.

Let's reconsider the example of the tests of the menu navigation items presence depending on the role of the logged in user.
At the moment that there are mistakes (too many or too few menu items), there can be different causes: de server can be down temporarily, the user is not logged in or does not have the correct role, or the menu ACL is incorrect or incorrectly implemented (the very point we wanted to test).

In such case it is practical to be able to take a screenshot of the page under test, before the assert and verify statements are executed.

In Selenese this works as follows:

captureEntirePageScreenshot

/screenshots/test1.png

background=#FFFFFF

In the Java tests I wrote the following function to ease taking screenshots:

protected void screenShot(int nr) {

    selenium.captureEntirePageScreenshot(

        screenshotdir

            + this.getClass().getSimpleName()

            + "-"

            + nr

            + ".png",

        "background=#FFFFFF");

}

In the above fragment "screenshotdir" is the path where the PNG files will be saved (with slashes in the correct direction depending of Linux/Windows).

My experiences as functional analyst

Wednesday, 26 February 2014

Learning to use Selenium in a project