Featured The QA Consultancy logo

We Are On The Move!

The QA Consultancy logoMost of you know by now, but I recently started a QA consulting firm, The QA Consultancy, that specializes in assisting companies and organizations with their QA strategy. I am not looking to do the testing itself, but rather help develop the correct strategy for them.

Due to this, I will no longer be posting to this blog, but rather will be moving all new content (and some of the old) over to the new blog page that is hosted on our main website. I would love for all of the wonderful followers of this page to head over there with me as well!

The QA Consultancy Blog

I am planning to keep this page alive for the older content, and also so that the QA Meetup posts have a place to live on.


Finding Efficiencies in Software Testing – Melissa Tondi

A huge thanks to Melissa for presenting last week!

Slide Deck
Finding Efficiencies in Software Testing – Melissa Tondi


Session Recording

Recap: How We Do QA: HomeAdvisor Automation

Sorry for the delay in getting these posted, but here is the slide deck that we went through in our last meetup.

Thanks again to everyone that was able to make it!


Let’s Build a Framework – Day 3

Introducing the API and Self Navigating Page Objects

Well we’ve reached the end of our little journey and in the course of 3 (and a half) sessions have built quite a handy little framework.   This post will sum up what we achieved in the last session where we added a new feature, self navigating page objects, and introduced a basic API test structure so we can run both our UI and API tests from the same framework.

Before we dig too deep like those little guys in that Lord of the Rings movie, lets take a high level look.

What is a self navigating page object?

When working in a page object framework you will find frequently you just need to get to a specific page, and you don’t care how you get there.  For instance, if you want to run negative tests on a form, you just need to get to the form, and don’t care the path or data used to reach the form.  This is especially relevant for smoke tests, where we just want to validate a page loaded correctly.

We are going to build a feature in to allow us to say navigate to this page I don’t care how.  We are going to do this by adding a generic navigateToPage method that all our pages will have, and each page will have something called a navigationPath that gives instructions on how to navigate to the page.

What exactly is involved in a API test framework?

For the second part of the day we will be creating a framework to allow us to run straight API tests along with our existing UI tests.  Because testing API’s is more programatic and straightforward our framework will be pretty easy to set up.

However, we will add one special piece of functionality called a detokenizer, that can take a API endpoint and magically give it all of the necessary parameters, and information to be a successful request.  This detokenizer can than be used to make much more impressive tests.

Creating a Self Navigating Page Object

Now that we know what our plan is, let’s get started.  First we will add the functionality to make all of our page objects self navigating.

Defining the Navigation Path

To conceptualize how to get to a page we can think of a set of instructions matching our current test layout.


So lets say we wanted to skip straight to the ebaySearchResult page, we can visualize this like so:

  • Step one
    • Page: ebayHome
    • Action: open
    • Data: none
  • Step 2
    • Page: ebayHome
    • Action: searchFor
    • Data: ‘orphans’

Almost all of our pages can have a similar list of steps.  So now that we know that we can define these steps inside our page objects through JSON:

Screen Shot 2017-01-16 at 11.56.41 AM.png
Defining A Navigation Path for Our Page Object

It’s that simple.  Notice that we are hardcoding the data for the 2nd step.  This is because the concept behind navigateToPage is that the test writer just wants to get to the page, they do not care how.

However, it is conceivable that in the future you would want to allow people to specify which data is used to reach a page.  This is 100% possible, but outside of the scope of this project I leave it as an exercise for you dear reader.

Creating a Navigate To Page Function

Once we have defined our navigation path for all of our page objects (or the page objects we think should have them) we need to have a function that all pages can use, that can read the path and turn it into actual test actions.

We’ll call this method navigateToPage() and because we want every page to have access we are going to create a new custom command for it.

Note:  We could also add this as a method on our PageObject base class to achieve the same effect.  The reason we don’t is because we don’t know if every page will have a navigation path so making it a command allows some pages to “opt out” of being self-navigating.

First lets see what the code would look like to take one step out of our path and execute it as a test step:

 var page = navigationStep.page;
 var action = navigationStep.action;
 var data = navigationStep.data;
//Same as on.<page>().<action>(<data>)
 return on[page]()[action](data);

It’s just that easy.  We have transformed our basic algorithm (below) into code (above)


But now we have a problem.  The above code works great for a single step.  But in a navigation path has multiple steps we are going to need to execute each step in the proper order.

We can’t use Promise.all like in other places in the framework because that runs all the promises at the same time.

Do not panic, creating the method will be surprisingly easy but we will need to use a new concept in promises: “each”.   Promise.each takes an array of data, and a function that returns a promise based off that data.

Note this is a big difference between Promise.all and Promise.each, Promise.all takes an array of promises, while promise.each takes an array of data and generates promises from it!

Lucky for us our navigation path is an array of steps. So to execute all of our steps:

 return Promise.each(navigationPath, function(navigationStep) {
     var page = navigationStep.page;
     var action = navigationStep.action;
     var data = navigationStep.data;
     return on[page]()[action](data);

Finally we throw it all into a single command and we end up with:

Screen Shot 2017-01-16 at 12.13.13 PM.png
Completed navigateToPage command

Seeing navigateToPage In Action

Now that we’ve built it, lets take it for a test drive.  You may remember this test we created on earlier:

it('Should open ebay and search for orphans', function() {

     return on.ebayHome().open()
     .then(function(result) {
          return on.ebayHome().waitForPageToLoad();
     .then(function(result) {
          return on.ebayHome().searchFor('orphans');




With our new navigate To Page we can shorten it to:

 it('Go to eBay Search Results page with navigation path', function() {
     return commands.navigateToPage(on, 'ebayCart');

Pretty amazing right?

Creating The Easiest Smoke Test Ever

Now we have self navigating pages, what better way to show the power, than by creating a smoke test where all we have to do is give a page name, and our test can navigate to the page and validate all elements are loaded.

To do this we can also take advantage of data driving in mocha.   To start lets define a list of pages we want to test in an array.

var tests = [
page: ‘ebayHome’,
description: ‘Home Page of Ebay’
page: ‘ebaySearchResults’,
description: ‘Search Results Page of Ebay’
page: ‘ebayCart’,
description: ‘Shopping Cart Page of ebay’

Notice, we added a 2nd field called “description” this field is going to help our test output be more human readable.

A nice feature of mocha, is you can dynamically generate “it” statements by using a basic loop.  So we can take our array of tests and using a forEach loop generate a new test for each page.


tests.forEach(function(test) {
    it('Make sure page ' + test.description + ' loads correctly', function() {



This will generate 3 tests from our array, one  for each page.  It will also use our “description” so our test output will be much cleaner.

Finally we all we have to do is

  1. Call navigateToPage for our page
  2. Call waitForPageToLoad after we’ve navigated
Screen Shot 2017-01-16 at 12.25.50 PM.png
That’s a Pretty Boss Smoke Test Right There

That might not seem like much but we have just created a way to validate every single page in our application.  Simply by adding an entry to our “tests” array.  We could even go one step further and dynamically grab all pages from our “pages” directory and do the same.  Meaning whenever a new page is added it is automatically part of the smoke test.

Pretty boss indeed … hoss.


Creating an Automated API test framework

Now on to the final piece of our automation framework.  We are going to add a little API testing framework to allow us to run both API and UI tests.  For the sake of brevity we are going to assume you are working with REST APIs and have a general understanding of what that means.

Note: The world of API testing is a big one, and this framework we are going to create is by no means comprehensive it is just a quick and simple one to allow you to get started and get additional coverage.

Starting Our Test Server

In order to test our APIs in a consistent way for all readers, this repo comes packaged with a test server that exposes 2 APIs we can use in testing.  To start the server use the below command from the main directory in the repo in a separate command prompt or terminal window.  Once started leave it running

node test_server/server.js

Our Test Endpoints

Below is a list of the endpoints exposed by our test server that we will be writing tests against

  • POST – /users
    • Takes a request body for a new user to be created
      • Request Body requires fields firstName, username, password, and ssn
  • GET – /users/:user_id
    • Returns a user that has the user_id passed in or returns a 404 if none are found

Creating a Basic Test Using Supertest

What we need to know

To get started first we are going to create a very simple test using the supertest-as-promised library in node.  Mainly because we are already working with promises in our other framework.  If you can’t stand promises there is a equivalent ‘supertest’ library that uses callbacks that you can research and use.

To execute an API we will always need these pieces of information:

  • verb
    • e.g. GET, PUT, POST, DELETE
  •  domain
  • path
    • e.g. /users/:userId/items
  • headers
    • e.g {ContentType: application/json, Accept: application/json }

Additionally for some APIs that use PUT or POST verbs we also need a request body.

Our first test will be to hit our POST /users endpoint without passing in a request body.   This should be an invalid request and we would expect to receive a response with a status code of 400.

Creating a Url Factory

To save ourselves a lot of code duplication firs thing we’ll want is to create a simple url_factory.js library and put it in our /api-framework/lib directory.  The url factory will be where we keep all the domains and paths we might want to hit in our tests.  It will do this by exposing methods.  These methods can take arguments so we can dynamically create paths or switch domain per environment.

First thing we want is a way to get our domain:

module.exports = {

    getApiDomain: function() {
        return 'http://localhost:3000';


Because we are using our test server our domain is always going to be http://localhost:3000 but this could change if we need to support multiple environments so you can imagine it taking a env variable and returning a different domain per environment.

Next we’ll want methods that will get us the path for our 2 endpoints.

getCreateUserPath: function() {
    return '/users';

getUserPath: function(userId) {
    return '/users/' + userId;

Now it may seem silly to create a method just to return ‘/users’ but we always want to think in terms of change.  If we have 100 tests that all use ‘/users’ in the test and one day our company changes the path to ‘/user’ without the ‘s’ we now have to update 100 tests.  Whereas with the factory we only need to update this method.

For the get users path we can now support passing in the userId and dynamically creating the path with the userId provided.

Creating A Basic Test

Now that we have a url factory we are ready to create our basic test with supertest.  Supertest is a library that uses a “builder” pattern and promises to execute a rest request.  The standard format of a supertest API call looks like this:

.then(function(response) {
   //Do your validation or other calls in here

In this case both headers and requestBody are optional.

Given this we now can create our test.  We’ll create a new test file in /api-framework/tests directory and call it api_tests.js.  We’ll need to require supertest, and for readability we will require it and store it in a variable called “request”

var request = require('supertest-as-promised');

Then we’ll create a new test:

describe('My first API Test', function() {

   //Basic API test for a get request with Supertest
   it('Should return a 400 when I make a request for a user without a request body', function() {

It can be helpful when working with APIs for the first time to just write the test using empty variables to make sure we have the structure right.  So lets do that using the supertest-as-promised structure discussed above

var domain;
var path;
var headers;
var requestBody;

return request(domain)
.then(function(response) {
    //validate here


That’s our test.  Now all we have to do is assign values to our variables.

  • Domain – We can use the url factory
    • domain = urlFactory.getApiDomain()
  • Path – We can use the url factory
    • path = urlFactory.getCreateUserPath
  • RequestBody – Our test is to test what happens when we send an empty one so:
    • requestBody = {};
  • Headers – We can hardcode for now
    • { ContentType: application/json, Accept: application/json}

Now our test looks like:

var domain = urlFactory.getApiDomain();
var path = urlFactory.getCreateUserPath();
var headers = {
   'Content-Type': 'application/json',
   'Accept': 'application/json'
var requestBody = {};

return request(domain)
.then(function(response) {

So you may we wondering why we are hardcoding the headers.  That is a very astute point.  If we had more time I would create either a common library or a headerFactory and have a method like getJsonHeaders().  But for now we are just hardcoding because they will be the same for every test.

Now we have the “execution” portion of our test knocked out.  The only thing remaining is a “validation”.  For this we are going to use an assertion library for Node/Mocha called “chai”.

Using Mocha Chai Assertions

“chai” is a library built to make english like validations.  To use it we need to add a require statement at the top of our test:

var expect = require('chai').expect;

This gives us access to the full “expect” library.  Once we have this we can do very pretty assertions like:

  • expect(<valueFunctionOrObject>).to.eql(<expectedValue>)
  • expect(<valueFunctionOrObject>).to.not.exist;
  • expect(<array>).to.include(‘<valueInArray>’)

With these pulled in we can add 2 validations to our test.  First we want to validate the statusCode is a 400.  Second we want to validate the error message returned meets our expectations:

 expect(response.statusCode).to.eql(400, 'Didnt get the status code I wanted with this url ' + domain + path + ' : instead got this body ' + util.inspect(response.body));
 expect(response.body.message).to.include('Could not create user because of error: Cannot create user without the following fields');

Understanding the Response Object

Looking at our validations we are referencing properties in the response object.  Let’s take a quick aside to understand what is included in the response, because it will dictate what we can validate:

  • response.statusCode – this is the status code returned by the api request
  • response.headers – these are the headers returned by the api request
  • response.body – this is the JSON body returned by the request (if it is not a JSON request do not use this)
  • response.text – this is the raw text returned by a request (for plaintext or xml requests use this)

The convenient thing here is that we can quickly parse any request body thats Json incredibly quickly.  In our case we expect a response body like

    message: 'whoa something went wrong'

So we can get the message simply through


The Finished Product

Basic Test Using SuperTest

Creating a More Dynamic Way to Test APIs

The above test is simple enough and you can use this format to write most of your functional tests without much hassle and they will be consistent and maintainable.

But let’s go one step further and just like our self-navigating page objects create a way to take any verb/url combo and have it create a ready to run test complete with valid information.

For example, to test the /users/:user_id endpoint we know we’ll need a valid user_id for successful tests.  To get a valid user_id we can imagine we’ll probably have to create a new user first, if we don’t create a new one we run the risk of having stale data or missing data if someone deletes our user.

You can probably imagine a much larger set of APIs where you need all sorts of data, the end result could be a very clunky and hard to read test.

Enter the detokenizer.

Creating a detokenizer

The concept behind a detokenizer is it can find any “token” passed in, and replace it with a valid value.  In this case we are going to consider a token to be anything like {user_id}.  So anything enclosed in ‘{‘ and ‘}’.

So if  I pass in a string of /users/{user_id} it would return /users/1234 or a valid user id.

The logic for this is pretty straightforward

  1. For Each token found in the string
    1. Go to a special library and find a method called "get_default_<token_name>"
    2. Call that method and replace the token in the string with the value returned

So what’s this code look like:

//Find all tokens in a string and replace them with valid values
 detokenizeString: function(tokenizedString) {
  return new Promise(function(resolve, reject) {
    //We clone the string to make sure we are returning a new string and not impacting the one passed in
    var newStr = _.clone(tokenizedString);
    var promiseArray = [];

   //match all tokens in the string
    var matches = module.exports.findTokens(newStr)
    if (!matches) {
       return resolve(newStr);
    promiseArray = module.exports.callDetokenizerMethods(matches);

   //wait for all our detokenize calls to complete
    return Promise.all(promiseArray)
    .then(function(results) {

      results.forEach(function(res) {
         //results come back in the form {token: '{tokenName}', value: 'value to use'}
          newStr = newStr.replace(res.token, res.value);
      return resolve(newStr);

Now thats a lot of code.  You may notice that its calling some other functions too.  Below is the code for the other methods as well.

callDetokenizerMethods, findTokens and computeMethodName

  • callDetokenizerMethods
    • Checks if there is a method called “get_default_<tokenName>” if so call that method and push it on the return array.  If not reject and complete to inform tester a method is needed.
  • findTokens
    • Uses a regular expression to find all tokens in a string
  • computeMethodName
    • take a token and return the expected method name
//for all the found tokens call the appropriate 'get_default_' method to get a valid value
 callDetokenizerMethods: function(matches) {
    var promiseArray = [];
    matches.forEach(function(token) {
       //Compute the detokenizer method name
       var expectedMethodName = module.exports.computeMethodName(token);
       if (typeof(detokenizerMethods[expectedMethodName]) !== 'function') {
         return reject(new Error('COULD NOT FIND METHOD FOR TOKEN ' + expectedMethodName));

    return promiseArray;

//Uses a simple Regular Expression to find all tokens in a given string
 findTokens: function(tokenizedString) {
    reg = new RegExp(/{(.*?)}/);
    var matches = tokenizedString.match(/{(.*?)}/g);
    return matches;
//Determines the correct method name to call to get a value for the token
 computeMethodName: function(token) {
    var expectedMethodName = 'get_default_' + token;
    expectedMethodName = expectedMethodName.replace('{', '');
    expectedMethodName = expectedMethodName.replace('}', '');
    return expectedMethodName;

Now in API testing its not just the URL that can have tokens.  Consider our POST api.  For it to be successful it needs a request body like:

   firstName: '{validFirstName}',
   username: '{validUsername}',
   password: '{validPassword}',
   ssn: '{valid SSN}'

Some of these can probably be hard coded like firstName, but some may need to be dynamic like “user_id” in the above example.  The good news is we can use the same logic (with a little fancy twist) to detokenize a request body.

  1. For Each key in the request body object
    1. If the key is a string call our detokenizeString method
    2. If the key is another object recurse and call our detokenizeObject method

The fancy “twist” being recursion.  If you are not familiar with recursion don’t worry we will paste the code for both of these functions further below, and they wont need to be updated.

Here is what that code looks like

//Takes an object and checks each field in the object if that field is a string we
 //call detokenize String to replace any tokens with valid values
 //If the field is an object, we recurse and call detokenizeBody again for the sub object
 detokenizeBody: function detokenizeBody(inputBody) {
    var self = this;
    //We use cloneDeep to ensure we return a new object and don't actually change the one passed in
    var body = _.cloneDeep(inputBody);
    var promises = [];
    var totalKeys = Object.keys(body).length;

    //if the object is empty no need to go further
    if (totalKeys === 0) {
       return Promise.resolve(body);

    Object.keys(body).forEach(function iterateKeys(key) {
       var val = body[key];
       var newVal;
       //if its a string call detokenize string to return a new string with all tokens replaced with a valid value
       if (typeof (val) === 'string') {
          .then(function(newVal) {
             body[key] = newVal;
       //if its an object then start over for the new object
       } else if (typeof (val) === 'object') {
    //once all tokens are replaced send back our fancy new request body 
    return Promise.all(promises)
    .then(function(result) {
       return Promise.resolve(body);

A Quick Apology

Now that is a ton of code posted above, and we don’t have time or space in this already long blog post to dig in and explain each line.  But there are no new concepts in the code we haven’t already covered.  And the code is well commented.   I highly recommend taking a look and seeing if you can figure it out.  But apologize for making you do so.

Creating business specific detokenizer methods

So in the code avalanche above we are trying to find out if there is a method called get_default_<tokenName> in a special library called detokenizer_methods.  This is how we get a valid value.  So for any token we want to cover with the detokenizer, we need to have an equivalent method in our new library.

In our test situation we need a method for user_id, username, and ssn.

These methods are very simple we just need to create whatever data we want and return it in this form { token: ‘{username}’, value: ‘validValue’}.  For username and ssn, we can do this very simply:

//Returns a guaranteed unique user name by using a date stamp
 get_default_username: function() {
    return Promise.resolve({token: '{username}', value: Date.now()});

//Returns a hard coded valid ssn, this might change in the future so we leave the method here
 //Even though this is hardcoded and not asynchronous we still return a promise because the detokenizer treats every method like it might be a promise
 get_default_ssn: function() {
    return Promise.resolve({ token: '{ssn}', value: '755-90-1324'});

Why promises?

We are using promises, because some tokens may require us to actually create new data to get a valid value while others can just use static data or easily created data like Date.now().

Our final token user_id, we are going to need to create a new user.  For this we will need a userHelper library.  This library will create a user using our own endpoints and return that user.  We can then use the userId returned for our valid token.

Here is what our user_helper.js library looks like:

Screen Shot 2017-01-16 at 2.16.52 PM.png
UserHelper createNewUser method

This code is very straightforward, just like our first test we are simply taking 3 arguments and using them to create a new user using our POST /users endpoint.  If the request was successful we return the response body which should have the userId, if not we reject and fail the test.

Now that we have the userHelper we can finish our final detokenizer_method for user_id

 //Creates a new user then returns that users Id to guarantee its always a valid user id for the test
 get_default_user_id: function() {
    //create a new user and return that users id
    return module.exports.get_default_username()
    .then(function(username) {
       return userHelper.createNewUser(username, 'glenn', '123-45-6789');
    .then(function(result) {
       return Promise.resolve({token: '{user_id}', value: result.id});

Now when our detokenizer calls get_default_user_id we go to our helper and create a new user, then we return the Id of that user to use in our test.

Seems like a lot of work to set up.  But from now on we can guarantee any test needing a user_id will have a valid one.  Also, going forward, we can add as many methods as we want for each token making it easier and easier to create tests for new endpoints

Putting it All Together

Let’s look at what our test looks like now using our detokenizer framework:

Screen Shot 2017-01-16 at 2.21.10 PM.png
Testing the GET request with the detokenizer
Screen Shot 2017-01-16 at 2.21.44 PM.png
Testing our POST request with a detokenizer


Where To Go From Here?

Now it may not seem like much savings with the detokenizer, but it is the seed from which you can do many great things with API testing.  Imagine you want to any of these tests:

  • use a invalid value for every parameter and test the response
  • use a missing value for every required query parameter or request body value
  • test a missing request body for every POST/PUT request

You can use the concept of detokenizing to test every one of these in a data driven fashion.  In fact the parent company of this post has a blog post covering a framework built exactly on top of this concept.  This framework is responsible for over 4000 tests dynamically generated and automatically detected whenever a new endpoint is added.

A fully formed Detokenizing API Test Framework

At Long Last

Well it’s been a fun 3-5 hour journey creating this framework with you.  Now that we are done, I encourage you to treat this framework not as a finished product but rather the start of your own fantastic framework.  Modify it to suit your company with your page objects and endpoints.  Create new features or change existing features to suit your needs.

It’s been said that great works of art aren’t created, they’re abandoned.  If you’ve followed along with this and created it yourself don’t abandon it now, keep going and see just how far you can get.  Then make sure to let us know.

Thanks to everyone!



Let’s Build A Framework – Day 2.5

Despite my best efforts we fell a little behind in the last 2 sessions.  So to catch us up to where I wanted to be on our final day we mae this Day 2.5 session post.  This post will cover a few key changes made after Day 2.   We will discuss these on Day 3 as well, but this post just covers the actual coding done.

I will keep this post as short and as sweet as possible.

What Has Changed Since Day 2

If you’ll remember on Day 2, we built a handy dandy function “waitForPageToLoad” that allows us to quickly check all required elements on a page and ensure that page is fully loaded.   However, the way we built it, was to add it directly to our EbayHome page object.  This means that only the EbayHome page could use it.  When what we want is for any page we create to be able to use waitForPageToLoad.

To do this we are going to create a Base Page Object and explore the concept of inheritance in javascript.  This will also let us clean up our code a little bit in other ways.

The other changes made were to create a couple new pages for us to play with, as up until now we’ve been doing everything on a single EbayHome page.  By the end of this post you should have 3 different pages and we can play with navigating between all of them and really start to see the power behind our framework.

Building a Base Page Object in WebDriver.io

As mentioned, our big goal for this session is to share methods between our page objects.  Now, there are plenty of options we could explore,  but, since we are already using classes and objects in our framework we should really take advantage of the features for them.

The biggest feature we’ll use is “inheritance”.  That is to say, allowing one class to be built on top of another class.  This means that the child class  would get all of the benefits and methods of the parent class while also being able to have its own methods.

Note: In computer science inheritance can be a controversial subject on when it should and should not be used.  If you are new to programming and want to learn more I recommend googling concepts such as “inheritance vs. composition”.   But for the purpose of our sessions we are going to stay away from these arguments and use inheritance in a very limited capacity.

Let’s Get To It

Our goal is to build is a new page object base class we’ll call “PageObject”.  This class will have a constructor just like our other page that saves our driver.  We will also use a little trick to have it load our locator file dynamically to save us some coding time.

But, most importantly we are going to put our “waitForPageToLoad” method there.  Doing this will allow any page object like EbayHome that inherits from PageObject to be able to call “waitForPageToLoad” with no extra code needed.

Building Our Base PageObject class

First step is we are going to make a very simple class, and only have it handle the driver.  This is going to look very similar to our existing EbayHome class.

Screen Shot 2016-12-08 at 12.54.03 PM.png
A very basic PageObject base class

So at this point with our EbayHome class inheriting our base class, all we’ve gained is it no longer has to manage its own driver.

Screen Shot 2016-12-08 at 12.55.58 PM.png
EbayHome inheriting our PageObject base class

First thing, let’s get rid of the need to constantly have to load the locator file in each page.   This is problematic because if we ever move our page object file, or change the locator name our tests will stop working.  Instead, we want our base class to be smart enough to say “Look for a locator file that has the same name as this class and load that”.

Luckily we can get the name of our EbayHome class even from inside our base class with


Calling that in our PageObject base class will return “EbayHome”.  So we can modify our PageObject constructor to automatically load our locator file for every page that inherits from it.

Screen Shot 2016-12-08 at 12.59.33 PM.png
Dynamically load the correct locator file based on the page objects name (EbayHome etc …)

Obviously now, we can delete the locator line from our EbayHome constructor.

Next up, we want to migrate our “waitForPageToLoad” method to our PageObject base class so for any new page object we create we can call this method.  This is fortunately very easy as we’ve already written the function.  We can just copy the waitForPageToLoad method directly to our PageObject class.

Screen Shot 2016-12-08 at 1.02.10 PM.png
waitForPageToLoad now in the base class

Note: the one thing we have to change is, its now a prototype of the PageObject class and not from the EbayHome class.

Now obviously this method relies on our helper method “findRequiredLocators” so we’ll have to move that over to our PageObject class as well.

Screen Shot 2016-12-08 at 1.03.59 PM.png
findRequiredLocators method now in PageObject class

Inheriting From Our PageObject Class

There are numerous ways to inherit in Node/JavaScript, in newer versions like ES6 its much more user friendly.  But because we want this post to work for everyone including people who might be on older infrastructure, we are going to use an older method of inheritance

I leave it to you the reader to make a modification yourself and use the new fancy methods of inheritance for future learning and practice.

For our method of inheritance we are going to use a helper method in the ‘util’ library called “inherits”.

util.inherits(EbayHome, PageObject);

That line will do most of the dirty work for us in linking EbayHome as a child of PageObject.

The only remaining thing is to use PageObject’s constructor.  So when we create a new page object in our tests we can pass a driver in to the page, but have that page pass it up the chain to its parents constructor.

Thankfully there is an easy solution for this.  We just call that driver directly from our parent class like so:

function EbayHome(passedInDriver) {
 PageObject.call(this, passedInDriver);
 locators = require('../locators/EbayHome');

Note that this is our EbayHome class, and inside we are referencing our new PageObject class and calling its constructor.

After these two changes are made, we have successfully made EbayHome a child of our PageObject base class in javascript.


And that’s it, nothing else needs to change.  You can still call waitForPageToLoad in your tests like before only now we’ve set it up so it will work for every new page we create as well.


Speaking of new pages ..

Expanding our Page Object Collection

Now that we have a few commands, a PageObject base class and some helpful methods.  It’s probably time to create some new page objects so we can make bigger scripts.

For this session we are going to create a EbaySearchResults and EbayCart page, and make a test to tie it all together.

Steps To Create a New Page

Obviously in Day 1 and 2 we covered making a new page.  But now is a good time for a refresher.

  1.  Create a new PageObject class under the pages directory

    1. Make sure that class inherits from our PageObject base class
    2. We’ve created a “SamplePageObject” class for you.  If you are in a hurry you can just copy this class for new pages and change “SamplePageObject” to your new page name
Screen Shot 2016-12-08 at 1.10.17 PM.png
A sample new page object template

2 Create a new locator file with the same name as the page under the locators directory

  1. Just like with the page object, we’ve created a SamplePageObject.json file you can use as a template
Screen Shot 2016-12-08 at 1.12.09 PM.png
Sample page object locator file

3 Add an entry to our PageObjectCollection class

  1. Now that we have a page, and a locator file, the final step to wiring it up is to add an entry to our PageObjectCollection class which is located in the pages/page_object_collection.js file.
Screen Shot 2016-12-08 at 1.15.34 PM.png
New entry for a page object in the PageObjectCollection class

4.  Fill out the new page with locators and methods

  1. Now you can add locators to the locator file for elements that are on the page.  And you can add methods in the page to interact with those elements.
  2. Below are screenshots of new locators and methods for our EbaySearchResults, and EbayCart pages!
Screen Shot 2016-12-08 at 1.17.27 PM.png
Locators for new EbaySearchResult page
Screen Shot 2016-12-08 at 1.18.20 PM.png
Locator for new EbayCart page
Screen Shot 2016-12-08 at 1.19.13 PM.png
New EbaySearchResult page with a goToCart method
Screen Shot 2016-12-08 at 1.20.01 PM.png
New EbayCart page

Putting It All Together With a New Test

Now that we have our shiny new PageObject base class, and some new page objects.  Let’s write another test, this time navigating further into the site.

Screen Shot 2016-12-08 at 1.22.00 PM.png
New test using new page objects

Note how we can call waitForPageToLoad on any page we want.

Hopefully now, you’ll start to see how this framework while kind of a pain to build adds some real value.

  • At this point our tests are readable even for people who can’t code well.
  • We can quickly spin up new pages in a matter of minutes and incorporate them into our tests.
  • We can add new methods to all our pages at once with our base class.

And, believe it or not, we have only scratched the surface with what is possible with our framework.

What’s Coming Up?

For our final day of work, we’ve still got some fun features to add.

  1. We’re going to create a concept of a “Navigation Path” that will allow us to quickly get to any page we want.
  2. We’re going to create a smoke test, that takes advantage of everything we have and that can test every single page we have in one simple data driven test.
  3. We’re going to create some API infrastructure and tests.


So stay tuned dear reader because we’ve saved the best for last.


Let’s Build A Framework – Day 2 Recap

Thanks to everyone who came out for the 2nd day in our current series around building an automation framework.

This one is audio only due to a miss during the session, but the issue should be resolved for future sessions. In addition, we are going to try and update with visuals as well if possible.

Sorry for the delay in getting this posted…totally slipped through the cracks!

Let’s Build A Framework – Day 2

Fleshing Out Our UI Automation Framework

Hello again to all the West Denver Test Engineers, or any wayward internet traveller who stumbled across this page.  Today we are going to go over what we built in Day 2 as part of our great framework building experiment.   The good news, is this blog post should be much shorter than Day 1, which had to include a lot of ancillary explanation of the tools we were using.   The bad news? were nearing the end of our journey.


So let’s get to it shall we?

What Are We Going to Build Today?

Today we are going to expand our framework with some new killer features like automatic frame handling in our tests.  And introducing a way to make our tests even more readable with a page object collection.  Finally we will close with adding a great new feature that will allow us to validate any page we write is fully loaded without ever having to write new code.

A way to clean up our test readability

Before we get started let’s revisit what our last test looked like at the end of Day 1.

it('Should open ebay and search for orphans', function() {
   //I want to open a browser
   //I want to load google
   var ebayHome = new EbayHome(driver);
   return ebayHome.open()
   .then(function() {
     return ebayHome.searchFor('orphans');


It’s a little cleaner, but if we imagine a test spanning numerous pages we’ll have to create each page object every time.

it('Should do lots of actions', function() {
  var ebayHome = new EbayHome(driver);
  var ebaySearchResults = new EbaySearchResults(driver);
  var ebayUserPage = new EbayUserPage(driver);
  // you get the idea

Not only is this not pretty, it exposes a potential maintenance problem.  Say at some point in the future, we decide we need to pass an additional argument to our page objects instead of just the driver.  In this case we would need to update every test and every instantiation of every page to support that.

A maintenance nightmare.

To solve this we will build out a PageObjectCollection object.   This will be a simple class that stores all our pages, so we can simply ask for which page we want.  Through some clever naming, this will also greatly improve the readability of our tests.

A way to handle frames

For those of you who have experience with older websites, or sites that use a lot of iframes you probably know that frames can be incredibly frustrating to deal with in UI automation.  The reason being that even though you can see everything on the browser webdriver treats each framew like its own window.  And you have to switch to that window to see the content.

The result is a lot of test code like this:

.click('some locator')
.frame('back out of content')

Wherein we constantly have to switch into and out of frames in our tests.  And if someone forgets to switch back out tests might fail leaving a wake of very confused test engineers trying to debug.

We are going to expand our locator file to allow us to indicate if this element lives in a separate frame, and enhance our commands to automatically change into and out of a frame.  This will allow testers to write actions in peace without ever having to think of frames again.  Truly a marvelous future.

A way to automatically be able to validate any page has loaded correctly

Finally we are going to add our most powerful feature to date.  We are going to add a way through the locator files to “Tag” elements as being required for the page to be considered loaded.   Then we will write a new method that will be shared for all pages that automatically checks for every element thats required.

This gives us the opportunity to write easy smoke tests that checks every page of the application for missing elements.  It also allows us an elegant way in our test to wait for the page to load before proceeding, without resorting to ugly waits or polling functions polluting our pristine test waters.

Getting Our Hands Dirty

Now that we know what we want to do.  We can get down to the nitty gritty and build it out.  Let’s start with the page object collection.

Introducing the Page Object Collection

As mentioned above the goal for the page object collection is to give us a way to ask for any page we want and give us a fresh page object for it.   We can then include and initialize the page object collection in all of our tests.   And if we are clever we can give it a name like “on”.  This will result in our test code looking like this:

.then(function() {
  return on.ebayHome().searchFor('orphans');

Much more readable.  The “on” makes it read like english, so even non programmers can get an idea of what the test is doing.

Creating the collection

We’re going to put our collection in the pages directory.  So to start we want to create a file called page_object_collection.js.   This is going to be a class, not a library, so it’s going to look a lot like our page object.

Screen Shot 2016-11-17 at 12.40.12 PM.png
Basic structure of page object collection
  • Note that we have to require all of our pages at the top of the file.
  • Just like a page we have a constructor that takes the driver in.  We save this driver so we can pass it to any page the user wants.
  • Finally we add a “prototype” function for every page.  In this case we made the function name camelCase to stick with javascript conventions

So if you had 10 pages you can add a method for each page.  Then in your test you can reference any page with the structure


Implementing the functions is easy as can be:

PageObjectCollection.prototype.ebayHome = function() {
   var self = this;
   var ebayHome = new EbayHome(self.driver);
   return ebayHome;
  • Here we use the same “self” convention just for good coding practices
  • We create the page object just like we would do in our test and then we return it.
  • If you are wondering, nothing being done here involves I/O so we don’t need to wrap anything in a promise.  We are simply instantiating a page, but that has no impact on the actual browser or WebDriver.io.

Our finished result (with a stubbed out function for another page we want to add looks like this:

Screen Shot 2016-11-17 at 12.46.49 PM.png
Page Object Collection Finished Product

Now that we have our collection, we should modify our test to take advantage.

Modifying the tests to use the collection

For now we’re going to stick with our “tests/better_testing.js” file.  But we don’t want to lose our old test, so we’re going to temporarily disable it, and create a new test underneath it.

You can disable a test in mocha by adding “.skip” after the it statement like so:

it.skip('Should open ebay and search for orphans', function() {

As mentioned in day 1, you can have multiple tests per describe block, so we can add a new test right underneath our first one.

Note:  be careful to ensure you are adding your new test after the ‘});’ that closes the it block.  A common bug is someone accidentally tries to add their test inside an existing test.

it('Should open ebay and search for orphans but better this time', function() {
  //Test will go here

Obviously before we can use our page object collection we have to define it somewhere, and because its a class we have to initialize it.  The first step here is require it:

var PageObjectCollection = require('../pages/page_object_collection');

Note again, because its a class and not a library we use Capital first letters to indicate to people.  This is a common Node code convention.

As we discussed, we want to store our collection in a variable named “on” for readability.  We can define on next to where we define our browser near the top of the test.

var driver;
var on;

Next we need to create a new instance of the collection and assign it to our “on” variable.  Since we need to pass the driver in on creating it, we should probably do this before every test.  (Since the driver is closed at the end of every test).

So near the bottom of our beforeEach we can do this:

  on = new PageObjectCollection(driver);

Now we are cooking with grease.  And we can re-write our previous test but this time using our collection.

 return on.ebayHome().open()
 .then(function(result) {
    return on.ebayHome().searchFor('orphans');

Much more readable.

Our final result looks like this:

Screen Shot 2016-11-17 at 12.57.22 PM.png
Improved test using page object collection

Note: we shrunk some of the file to fit it all in the screenshot, don’t worry we still have our afterEach and first “it” test they are just minimized.

One goal down for the day 2 to go

Handling Frames Through the Locator File

Frames are a tricky thing, some engineers can go years without ever running across one while others must deal with them constantly.  Still building support for frames in your framework even if you don’t ever use a frame, could pay off down the line if you want to embed youtube or other site content suddenly. You’ll be able to support it with no effort.

To support frames we want to make it an optional field in our locator file.  Meaning if an element lives in a frame we want to add a field called “frame”:

 "topSearchBar": {
   "description": "Search bar at top of the page",
   "locator": "#gh-ac",
   "frame": "contentFrame",
   "groups": ["requiredForPage"]

Thats it for the locator file.  Now we want to modify our custom commands, to check if a locator has a frame, and if so to switch into that frame before running the command.

We’ll start with our “click” method.

Here we’ll do some actual node coding.  To check if a field exists in an object we just do

if (locatorObject.frame) {
//locatorObject has a property or function called frame

Then we will take advantage of the WebDriver.io chaining ability and just add a method to switch to the frame before our click, and one after to switch back to the main frame.

 return resolve(driver.frame(locatorObject.frame)

note that we keep the entire chain inside the resolve statement to ensure our promise works.

Then, we handle the case if an element does not have a frame by adding a else statement. In which we call click as we did before.  The whole thing looks like:

Screen Shot 2016-11-17 at 1.50.19 PM.png
Updated click command with frames

And we can update setValue the same way

Screen Shot 2016-11-17 at 1.51.55 PM.png
Updated setValue command with frames

And that’s all there is to that.  Both methods will now handle switching into a frame for any element that requires it, and switches out when done.

Building Out the “waitForPageToLoad” Feature

Now we’ve come to our last new feature we want to add as part of Day 2.   The feature of waitForPageToLoad.  Building this we’ll get to learn some new concepts about promises, and programming through loops.

Adding a tagging system to our locator files

The first thing we want to do is to be able to tag elements in our locator file to indicate this element should be present before the page can be considered loaded.  To do this, we are going to add a new field called “groups”, this is going to be an array where we can put whatever group we want this element to be a part of.

Our group is going to be “requiredForPage”.

Why make it an array?  Because we might find other ways and reasons we want to group elements so in thinking ahead we want to support it right off the bat.

Since we only have 2 elements so far, and both should always be present on the EbayHome page we’re going to mark them both as required like so:

 "topSearchBar": {
 "description": "Search bar at top of the page",
 "locator": "#gh-ac",
 "frame": "contentFrame",
 "groups": ["requiredForPage"]

 "topSearchButton": {
 "description": "Search button at top of page",
 "locator": "#gh-btn",
 "groups": ["requiredForPage"]


Add a new custom command to wait for a single or group of elements to be visible

Our next step is to add a way to tell if an element is visible.  If we want to wait until all elements are loaded we’re going to need this.  As an added benefit these methods will be available for future uses as well.

We’ll build these by going off the existing “waitForVisible” command in WebDriver.io.  And we’ll add our own waitForVisible methods in our commands.js file.

Luckily the code is pretty much identical to the code for click, the difference being we change our console output and we call waitForVisible instead of click.


Screen Shot 2016-11-17 at 2.23.21 PM.png
New waitForVisible command

Note: because the code is so similar you can get away with copying and pasting, but I highly recommend you re-type it out.  First for that muscle memory learning, and second to encourage us to find ways to avoid duplicating code.   We don’t have time in this framework but we could build a simple helper that we just pass our method to that avoids all the duplicated code.

Now that we can wait for 1 element, next is we want to wait for a group of elements.  This will be a little more complicated.  We’re going to want to create a new command called

 waitForGroupToBeVisible: function(locatorGroup) {


We’re going to assume someone has already compiled a group of elements into an array for us so we take a locatorGroup as an argument.

So we’ve done well with promises so far.  But in situations like this where we need to do multiple promises in a loop situation we need to use a special function our promise library has called “all”.

The “all” method in our library will take an array of promises, and run them all, and when it is finished it will return its own promise, with an array of results.  If any of the promises fail, it will fail as well.  The bonus is it runs through all these promises at the same time, which saves our program time as well.

var promises = []

.then(function(results) {
    //At this point we know all promises are done

Above is the basic form of using “all” for promises.

So we are going to use the same concept.  First we create an array to hold our promises

var promises = [];

Our next step is we want to loop through every locator in our locatorGroup and we’ll want to check each one if its visible.

locatorGroup.forEach(function(locator) {
    //Check if this locator is visible and push on promise array

This is an example of a “forEach” loop a very nice tool in Node/JavaScript.   It is a function that is called for each element of an array and for each element it will execute the function passed in.  Here it passes us a locator.

Our next step is to wait for it to load, which, lucky us, we just happened to write a function that does that!

locatorGroup.forEach(function(locator) {

So here were using a feature of arrays in javascript that we can add elements to an array with the “push” method.  We’re also using a feature that lets us call other functions  in the same library.  You can do this by appending module.exports.<functionName>.

So we basically go through each element call our waitForVisible method and store them all in an array of promises.  It’s important to note at this point in the code none of the promises have resolved they are all working at the same time.

So our final step is to wait until all promises are done.  Enter our “all” function.

return Promise.all(promises)
 .then(function(result) {
     return result;

As we discussed in the last day, that “return” statement is critical, if left off the method will end before any promise has finished.

That’s it we can now wait for a single or group of elements!

New waitFor visibility elements

Write our waitForPageToLoad method

Now all thats left is to add a method to our page, that will find all the elements that are required and call our new custom command.

For now we are going to add it directly to our EbayHome page, but in a future post we’ll talk about how to make this method available to every page by default.

So we are going to edit pages/EbayHome.js and add a new internal function to findRequiredLocators.  And to add a new public method called waitForPageToLoad

var findRequiredLocators = function(locators) {


EbayHome.prototype.waitForPageToLoad = function() {


So what the hell is going on here, why do we have this new way of defining a function?  When we define a function through “var myFunction = function” we are limiting the scope of that function.  People using our Page Object in a test will not be able to access our findRequiredLocators function, but they will be able to call our waitForPageToLoad function.  Because we added waitForPageToLoad as a prototype of our class.

However waitForPageToLoad will be able to call findRequiredLocators because they are in the same scope.  It’s crazy but it will eventually make sense I promise.

Lets start with findRequiredElements.  In the page we already have all the locators stored in our “locators” variable.  But woe be unto us, the locators variable is not an array, its an object, with each property referencing a element.  Because of this, we cannot use forEach to loop through, because thats for arrays.

Instead we are going to use a helper library called “lodash”.  We store lodash in a variable called “_”.  Making sure to pull it in from a require statement.

var _ = require('lodash');=

LoDash has a very handy method called “forOwn” this will loop through all the properties of an object like a forEach loop.  It will give us two values (key and value).  The key is the name of the element and the value will be the locatorObject.

var returnArray = [];
_.forOwn(locators, function(locator, keyName) {

    //Now we can check if a locatorObject is part of the requiredForPage group

To  check if a locator in our loop is part of the requiredForPage group is pretty simple

if (locator.groups) {
    if (locator.groups.indexOf("requiredForPage") !== -1 {
        //found a match

So first we check if it even has the groups field, if it has no groups we know its not part of requiredForPage so we can do nothing.

Next we use a feature called “indexOf” this will return the index of an array that matches, if it can’t find a match it will return a -1.  You can think of this as saying “if locator.groups contains “requiredForPage””.

So we are saying if it doesn’t equal a -1 it means this locator does have this group.  In which case we push it onto our result array.

Finally when were done we just return our array, which now contains all elements that are part of the requiredForPage group.

New findRequiredLocators helper function

Once we have this method, our waitForPageToLoad method is trivial.  We just find our elements and pass it to our custom command.

Screen Shot 2016-11-17 at 3.03.38 PM.png
Our official waitForPageToLoad method


That’s it!  It took some work, but from now on anytime we want to check if a page is ready we can call this function, and any new page we can tag which elements are required with a few keystrokes.

For our final piece lets see how this looks at our test level.  We’ll go ahead and modify the new function we added earlier …

Screen Shot 2016-11-17 at 3.05.26 PM.png
Beautiful Isn’t It?

What’s Next?

So now hopefully we are starting to see some of the benefits of our framework.  We are going to do a special day 2.5 session next.  Which will exist outside of the meetups and only on this blog.   In that we are going to do one quick stop gap fix to create a base class that all of our page objects will sit on top of.

This is going to let us use that waitForPageToLoad method on all pages.

After thats added, for day 3, we will make some finishing touches on the UI framework.   And we will switch over and build a quick API Testing component to allow us to test APIs as well.

Truly we live in interesting times.  If you made it this far I commend you, and keep your ear to the ground for the next post which should come in a week or so.



Let’s Build A Framework – Day 1


Building Out the Bones of Our UI Automation Framework with WebDriver.io


Hello to the West Denver Test Engineers, or to anyone out there who’s following along with our grand framework creation experiment.  This post, while very belated, will cover exactly what we did on Day 1.   We will talk about how you can check out Day 1 and see the end result.  Or you can check out Day 0 and follow along with the video posted earlier on this blog or you can follow along with this very blog post.

For this post we are going to assume you already read through the Day 0 Post and are set up and ready to create.  We will start with discussing the goals for the day, and go into how we build it.  Then, finally, we will discuss the purpose of what we built and potential uses for it for Day 2 and beyond!

The What: What We Want to Build Today

This section will cover what coding we actually did on day one, with pictures and more.

Ultimately we want to build a “Page Object Framework” which is a very common design pattern for UI Automation.

A Quick Understanding A Page Object Framework

The basic concept is we can divide any website or application up into individual pages.  We can sometimes further divide up that pages into smaller pages (think headers and footers).

Each page then has its own object associated with it.  This object stores the locators for all the elements on that page, and actions that you can do on the page.  This abstract breakdown of the application helps tremendously in writing easy to read and easy to maintain tests.

Think of a situation where you have 100 tests that all require a user to login.  Then one day a developer changes the login page.  You might find yourself in a situation where you have to update 100 tests.  Whereas if you stored all the details of a login page in a single object that the tests share.  You only need to update that one object and all your tests will magically work.

There are numerous components to build a successful page object.  Today we are going to focus just on this:

  1. Build a basic script with no framework to understand the basics of mocha, node and webdriver.io
  2. Create a locator file that stores information about what elements we want to interact with on a page.  (Stuff like buttons and text fields)
  3. Create a custom command library that can take on of our locators from our file and pass it through to WebDriver.io.
  4. Finally create a page object, which will be a class to represent a page in the application.  This will have easy to call methods representing actions you can perform on a page.  Actions like “login”, “searchForSomething” etc.

The How: Building It

Getting Set Up

Before we can start coding our framework we have a little foundational work to do to install, and set up our basic folder structure.

Please refer back to the Day 0 post for more specifics on everything you need to install.

Clone and enter our repo from git
git clone https://github.com/mustarddemon/westdenvermeetupfw.git
cd westdenvermeetupfw

Install and build everything
npm install
Start the selenium server (In a separate terminal)

In a separate terminal run the command for windows or mac, and leave it running.  When running tests for the remainder of this project we always want to make sure the server is running.


Create our folder structure

The final set up piece will be to create a series of folders that we will use to divide our code up logically.  The structure is a main folder to hold our framework “mynewframework” and inside that folder create sub folders “lib”, “locators”, “pages” and “tests”.

Code below is for mac/linux but it assumed windows users can create their own folders

mkdir mynewframework
cd mynewframework
mkdir lib
mkdir locators
mkdir pages
mkdir tests

And that’s it, we’re ready to start building our framework!

Creating A Basic Test Without A Framework

The first thing we’ll want to do is just create a straightforward WebDriver.io test with no framework.  This is important to understand the basic structure, and also to serve as a reference to the advantages of our framework as we get further along.

Understanding Mocha

For our tests we are going to be using the mocha test runner.  As such were going to need to understand the basics of Mocha.  And how to set up a test file.

Mocha (for our purposes) is a behavior driven test framework.  It uses concepts such as “describe” to group a series of related tests together, and “it” to define a specific test.  This is so you can get test output like this:

Ebay Home Page – Should be able to search for an item

The way this looks in code form is like this:

Basic Mocha Test Structure
Understanding Basic Mocha

Looking at the above example you can see a pattern, where the first argument to a “describe” or “it” block is the description.  Each describe block can have multiple “it” blocks under it.  It can also have additional “describe” blocks to further group tests together.

The 2nd argument is whats called an anonymous function, this is a feature of the JavaScript language which Node is built on top of.   You can think of this function as the actual work that will be done.  Note the bottom two lines of the code above, the first closes out the “it” statement, and the 2nd closes out the “describe”.  The ‘}’ ends the anonymous function, and the ‘)’ ends the call to “it” or “describe”.

If you are coming to this having worked in a compiled language like Java or C# this can look a little strange.  But right now we can treat this as necessary “code  noise” over time it will make more sense.

Before and After Blocks

Mocha like most test runners also supports using Before and After blocks.  This allows to have some code that runs before the entire suite “before” or before each individual test “beforeEach”.  The way this looks in code is:

Mocha with a beforeEach method
The “done” function

The final aspect of mocha we need to be familiar with is the “done” function.  Node by its nature and mocha as an extension are asynchronous.  That means that whenever I/O is encountered (like reading a file, interacting with a browser, or calling an API) it is placed on a stack to be executed while control of the program continues on.

This asynchronous nature while making for very fast program execution, can be difficult in testing.  And since for UI testing almost everything done in the test is I/O the test can sometimes complete erroneously before the browser has even opened.

Mocha offers a couple solutions to this, one is that if a test is written using promises, it wont finish until the promise has resolved (this will be discussed in more detail below).  The other is a “done” function which serves as a “callback”.  What this means conceptually, is that a test is not considered completed until the “done” function is called.

The “done” function is passed in as an argument to any “it” or “before” or “after”
statement (which means any test).  Then inside the test the writer of the test can call “done” at a point when they know the test has completed.

In code this looks like:

Using done to ensure test completion

Note that done is passed in to our anonymous function, and called when we are sure the test or before actions are done.

You may be asking, who is passing a a “done” function in, and what is it doing inside that function.  The answer is that mocha is passing a function in.  That function determines test results and what to do next.  So when you call “done()” in your test, you are telling mocha to execute its end of test logic and move on.

Writing Our Basic Test File

Now that we have a basic understanding of mocha, we can write our first basic test file.  Our initial structure looks like this.  If you are following along at home, create a new file called “basic_scripting.js” under the “tests” folder.  And type this in.

Why Type?  You’ll notice that most code here is in an image format.  This is intentional to prevent copy/pasting.  By typing out the examples you will build muscle memory and it will immerse you in the learning experience.  Studies conducted by me, have shown people learn much faster when typing out the code initially.

Screen Shot 2016-11-16 at 2.57.53 PM.png
Initial structure for basic WebDriver.io test

Just like we discussed above, but we added one new thing a “require” statement.

Require Statements in Node

This is how we import libraries or classes in node.  Whether they are our libraries, or from a third party or build in libraries.  The behavior to pull one in varies slightly

Libraries Built By Us

//We need to pass the relative path from where we are currently
var myLibrary = require('./lib/myLibrary.js');

For our libraries we have to pass the path to the library.  This path is from the file we are currently in.  So in the above example if we were requiring this library from a file in the tests directory we’d have to change it to “../lib/myLibrary.js”  the “..” indicating we need to go up a directory.

Native or Third Party Libraries (Not Built By Us)

//We don't need to pass the path to the library just the name
var libraryName = require('libraryName');
Initializing WebDriver.io

We are now at a point where we can actually start a browser.   Below is the basic scripting test we will write.  We will break down whats happening below the image.

A basic vanilla WebDriver.io test

So what’s happening here?

  • First we define a variable object called “options”.  This object has a property called “desiredCapabilities” which itself is an object.  The “desiredCapabilities” object has a property called “browserName” which we define as firefox.  This is how we set the browser for our tests.  This options object is what WebDriver.io needs to get started
    • One of the advantages of Node/Javascript is the ease in which we can create new objects.  We can quickly create objects within objects, and can even give these objects functions!
  • Next we start by returning a command to webdriver.io.  If you’ll remember we pulled webdriverio in from our require statement at the top of the test.
    • We return as a way to stop the test from completing.  The test won’t complete until the entire return statement is finished.  In this case you can see we are “chaining” together multiple commands which finished at “end()”
  • The next command in our chain is “init()” this is where the browser is actually started.
  • The first command in our “chain” is “remote” this is telling webdriver to connect to the selenium server that we have running and use that to start our browser.  We pass in the “options” object, which tells it which browser we’d like to start.
  • The next command is “url(‘http://www.google.com&#8217;)” this is a WebDriver.io method that tells our browser to open that webpage.
  • Our final command is “end()” this tells the WebDriver.io and the selenium server that we are done and the browser can be closed.  This is also the end of our command chain so we terminate it with a “;”

That’s it we’ve created our first test.  You can explore all the different methods you can use with WebDriver.io here.  If you were so inclined you could start writing tests right away.  You know how to set up a test file and how to interact with webdriver.  You can call methods like “click” and “setValue” to type and click things on your web page.

Note: In truth you can get very far with just basic scripting.  The reasons for a framework are not immediately apparent.   Issues with basic scripting tend to happen months down the line after you’ve build your 100 tests and now you need to change something.  I encourage anyone to write a few fully scripted tests using this basic method.  It will not only help your understanding, it will help explain the features we are trying to give in our framework.

Wait?  How Do I Run the Test?

Ah, a key oversight!  To run the test use the following commands (after making sure the selenium server is running and you have run npm install).

Make sure you are in the mynewframework folder where we created our folder structure

mocha tests/basic_scripting.js

Creating a Locator File

It’s difficult to pick a starting point when building a new framework as many of the components tie in together.   That being said, a key feature of our framework is going to be storing a lot of knowledge about our pages in external language agnostic files.

This has numerous advantages which we’ll explore as we continue to build out the framework.

We are going to be storing our locators in JSON files.  This is because its easier to write than XML and its got native support in JavaScript/Node which will make parsing it in our code much easier.  But, in truth you could use any form of external storage for your locators.

Our basic layout is going to be an object, and in that object each property represents an element on the page.  For our example we’re going to use ebay.com.

First, we’ll create a new file called “EbayHome.json” and put it in our locators folder, then we’ll identify a few elements on the eBay home page we know we’ll want to interact with in our tests. In this case we’re going to use the search bar at the top of the page, and the corresponding search button.   Here’s what our file will look like initially.

Screen Shot 2016-11-17 at 9.39.47 AM.png
Basic locator file layout

You can see we give each element an easy to read name, and inside the object for the element we are going to store data on how to locate the element.  We are also going to include a “description” field.  This gives even more information about the element, and will allow us to make some very elegant test output.

In this experiment we are going to gloss over how to locate an element.  We are going to assume all elements are located through css.  But in later sessions we’ll discuss how to easily expand this to support xpath or any other locator method.

Here’s what our file looks like after we add information for our elements:

Screen Shot 2016-11-17 at 9.42.34 AM.png
Complete locator file

And that’s it.  While it may seem simple this locator file will allow us to add incredible features which we’ll discuss in future sessions.

Creating Our Own Custom Commands

Since we will be using our own locators pulled from our locator files.  We don’t want to use the straight WebDriver.io methods like click and setValue.  We instead want to make a wrapper method that takes our version of a locator and passes the correct locator information to the native WebDriver.io method.

We’re going to store all of our custom commands in a file called “commands.js” which we will put under the “lib” directory.  This file is going to be a library, that means its just going to be a collection of methods we can pull into any of our code with a “require” statement and call those methods.

To help make our code readable and understandable we’re going to use a NodeJs concept called “Promises”.  We’ll discuss further below what promises are and why we use them.  But our initial commands library will look like this:

Screen Shot 2016-11-17 at 9.49.26 AM.png
Basic custom commands library layout

There’s a lot happening here so lets break it down.

  • First we require “bluebird” this is a third party promise class that we will use.  Note that we store the required class in a variable called “Promise”.  The capitalization there is intentional, in Node the standard is when requiring a library that we use camel case, but when we require a class we use Uppercase.
  • Next we define a variable called “driver” this is going to hold the webdriver.  You can consider the driver to be essentially the browser, and it is the Native WebDriver.io code.
  • Next we create module.exports, this is how we tell Node which of our methods are public.  Anything inside module.exports you can reference when you require the library.
  • Inside module.exports we define a setDriver method.  This allows us to pass the driver in once and reuse it in all of our methods.  This will make our code more readable.  Notice that we take a passedInDriver, and assign it to our driver variable.  The nature of Node is that once something is “required” all subsequent “require” statements load the same library.  Meaning once we set the driver it will stay set even across different tests.
  • Finally we define some “stub” functions for click and setValue.  These both take a “locatorObject” in, this is going to be one of the elements from our locator file.  Now we can implement our commands.

How To Call A WebDriver.io Function

Before we can implement our custom function let’s look at how we would call it normally.

driver.setValue('locatorString', 'text to type');

This is the base way to call a WebDriver.io method.   So we know our locatorObject has a locatorString property so we can modify this to be:

//Our setValue function has a 2nd argument which is the text we want to input
driver.setValue(locatorObject.locator, textToType);

So far so good.  But lets take advantage of that description field we put in our locator object.   We can make it so every time our custom function is called it outputs a nice description so anyone running our tests can see in English what is happening.

click: function(locatorObject) {
 console.log('I am clicking ' + locatorObject.description);

At this point our function will work, but using it will require tests to chain all methods together due to the nature of WebDriver.io.  We want to use promises to make our test code easy to write for anyone who might not have a lot of programming skills.

When using promises, we can wrap any non promise method in a promise and have it return that promise.  To do this we we immediately return a new Promise object.

return new Promise(function(resolve, reject) {
   //our method here

So what does this mean?  It means we immediately return the new Promise object, and we pass that object a new anonymous function.  That function takes itself 2 arguments resolve, and reject.  Both of these are also functions.  When we are sure our method is complete successfully we would call resolve, if an error occurred somewhere we would instead call reject.

This is how promises can enforce sequential behavior when dealing with asynchronous actions.

So we know we want to call “resolve” when our webdriver method is complete.  Luckily for us its as simple as:

return resolve(driver.click(locatorObject.locator));

That’s it.  Code using our promise wont move on to the next step until resolve is called, and resolve won’t be finished until the WebDriver.io click method is finished.

Putting it all together we get our finished commands file:

Screen Shot 2016-11-17 at 10.11.23 AM.png
Finished custom commands library

Now whenever we want we can expand our commands library to cover any WebDriver.io method.  Meaning any normal method you will call will now be able to have a nice output and use our locator objects from our locator file.

The advantages of this aside from the output are, that whenever we want to add a new feature to our locator object, we only need to modify these functions to support the new feature we don’t have to update anything else.

Some great features we’ll be adding later are:

  • Automatic handling of frames
  • Switching locator types between css and xpath
  • Grouping elements for automatic page load validation

Other features you could add

  • Alternate locators (great for mobile applications or applications where the locator changes between versions or users)
  • Automatic retry on failed actions
  • Checking for pop ups on failed actions (great for sites with ads)

Creating Our First Page Object With Locators

We’re so close now!  The final component of our framework is going to be a Page Object.  We are going to have a 1:1 ratio for page objects and locator file.  Meaning we need a page object called EbayHome.js to match our locator file EbayHome.json.  We’ll put this file in our pages folder.

This will allow us to automatically be able to reference all elements on a page.

The next thing we want our page objects to handle is what actions we are going to perform.  These actions are at a larger level then just click.  Think of actions like “logIn” or “searchForItem” or “waitForPageToLoad”.

Let’s start with a basic definition of our page.

Screen Shot 2016-11-17 at 10.19.53 AM.png
Basic page object class layout

So, again, a lot happening here let’s break it down.

  • First we require our handy Promise library like before
  • We also pull in our commands.js file we just made
  • And we define a “locators” variable.  This will store all the locators from our file.
  • Then we use the kind of funky way JavaScript defines classes.  It does so by defining a function with the class name.  This is the same as a “constructor” you would see in other languages.
    • Our constructor takes a driver in and assigns it to our driver variable.
    • It also loads the locator file and stores it in our locator object.
  • Then we define some “public” methods for our class to have.  to do this we use “prototype” syntax followed by the method name.  Here is where we will implement our actions.
  • Finally we set module.exports to our class.  This is what will allow us to create new pages.

Note: unlike libraries, classes in javascript need to be created with a “new” command, and doing so will create a fresh copy each time.  Meaning that each time we will have to pass a driver in and anything changed in the class will not be preserved across instances of that class.

Now for the fun part actually stringing together some actions to do something on the browser.

First we want an “open” method, this will load the url for the home page.  Unlike click and setValue we havent implemented a custom command for open yet. So we’ll just wrap it in a promise like before:

EbayHome.prototype.open = function() {
   var self = this;
   //open the ebay home page
   return new Promise(function(resolve, reject) {
      return resolve(self.driver.url('http://www.ebay.com'));

So what’s the deal with the “self” variable?  JavaScript does some interesting thing with scoping, so its just a good practice to create a variable called self and assign it to “this”.  I won’t go into more detail you can just think of it as some necessary code noise.

The rest of the method is just like our custom commands, we just wrap the built in WebDriver.io method in a promise.

The next method “searchFor” we get to use our custom commands though:

EbayHome.prototype.searchFor = function(textToSearchFor) {
   // type something and search for it
   var self = this;
   return commands.setValue(self.locators.topSearchBar, textToSearchFor)
   .then(function() {
     return commands.click(self.locators.topSearchButton);

So you can see we can call any of our custom commands just by using “commands.<method name>”.

Another nice feature here, is because we know our commands methods are already returning promises, we don’t need to wrap them in one, we can just return the method.

So what about this “then” business?  This is the key part of why we are using promises.  Anytime you are working with a function that returns a promise, you can call call “then” and pass a function to it.  That function passed will only be executed after the previous promise is resolved.  This means we can guarantee our actions happen sequentially.

This is called a “promise chain” meaning we can string together however many actions we want and have them execute in sequence.

//Typical Promise Chain Structure
return somePromiseFunction()
.then(function(resultFromFirstFunction) {
    return someOtherPromiseFunction();
.then(function(resultFromPreviousFunction) {
    //and so on and so forth

If you are new to Node this may seem strange, but you might still see the value of syntactical structure of the code.   “Do something THEN do something else THEN etc …”

Note:  The return statements are absolutely critical with promises.  Because of the nature of Node, if it sees a return statements it stops execution in that method until all promises are resolved.  If you left out a “return” Node would asynchronously run the promises and continue execution to the next command.  Meaning it would get to the end of the function and finish before the promises had finished.

Let’s look at our finished product:

Screen Shot 2016-11-17 at 10.38.00 AM.png
Finished page object class

Now we can create as many Pages as we want to map out the entire eBay site, and we can make as many actions on each page.  So that when it comes time to write tests all the hard work has been done for us!

Tying It All Together With A New Test

We’ve reached the promise land, time to write our first test using our framework.

Note: Be prepared to be underwhelmed, our new test will not be that much cleaner, but remember this is just day one.  In the coming days we will add more features to make our tests cleaner and easier to run.

We already have our “basic_scripting.js” file in our “tests” directory.  Let’s create a new file for our new framework style test called “better_scripting.js”.  We’ll give it a basic mocha structure:

Screen Shot 2016-11-17 at 10.41.32 AM.png
New test file layout

A couple things to point out before we go further.

We added a beforeEach and afterEach method.  We are going to use these to automatically start our browser and shut it down when the test is done.

We also added a “this.timeout(30000)” this is a feature in Mocha.  By default it times out a test after 2 seconds, so we are expanding it so it will kill our tests after 30 seconds instead.

Finally we added some new “require” statements we are pulling in our EbayHome class.  This is going to allow us to create a new version of the page and call our handy methods.  And we pulled in our custom commands, this is so we can set our driver in it at the start of our test once we’ve initialized it.

First though, lets move our initialize webdriver code into our beforeEach method.

 beforeEach(function(done) {
   var options = {
     desiredCapabilities: {
       browserName: 'firefox'

   driver = webdriverio.remote(options)
   return done();

So this is essentially the same code as in our “basic_scripting.js” but we call the “done()” method when we are done to ensure we don’t start the test before the browser loads.

Our “afterEach” method is even simpler.

afterEach(function() {

That’s it, it will close the browser after each test.

Now to the big moment.  We can write our new test using page objects:

First we want to create a new EbayHome page so we can use it.

var ebayHome = new EbayHome(driver);

Then we can call and chain any methods on the page together:

 return ebayHome.open()
 .then(function() {
    return ebayHome.searchFor('orphans');

Now we only have 2 methods, but you can imagine if you had multiple page objects, and methods you could write a pretty impressive test.

Lets see the finished product:

Screen Shot 2016-11-17 at 10.56.47 AM.png
Finished test file

Wait?  How Do I Run the Test?

Ah, a key oversight!  To run the test use the following commands (after making sure the selenium server is running and you have run npm install).

Make sure you are in the mynewframework folder where we created our folder structure

mocha tests/better_scripting.js


It’s still not super pretty and easy to use.  But by the end of day 2 all of that will change!

So stay tuned, and watch this space!

Let’s Build A Framework – Session 1 Recap

Great big thanks to everyone that came out and joined us for our kickoff session around building frameworks for automation. As mentioned, we were able to record the screencast of this session, and I wanted to get this posted for everyone to take a look at and view (or review).

GitHub Page

Meetup Page

Let’s Build a Framework – Day 0

Building An Automated UI and API test framework

A big hello to anyone from the West Denver Test Engineering meetup who came to this page in preparation of attending or watching the 3 part discussion on building an Automated Testing framework for API and UI testing.  And to anyone who has just stumbled across this page a big hello to you as well.

For this blog post we will focus mainly on getting set up prior to the meetup or prior to watching the videos to build your own framework.  We will talk about what tools we will be using, and what you need to install.  In preparation for this I have created everything locally to ensure it works, but have done so on a Mac.  We will include information on how to install and run on a Windows machine as well, however these instructions have not been tested as of yet.  As we get feedback I will endeavor to keep this up to date with the most recent information on tools and installation for both Windows and Mac.

If you’re a Linux user, then you are probably used to trudging up the hard road so good luck to you.  Actually, in all seriousness the steps for the mac should be very similar for Linux users so try to follow those.

The Tools We Will Need

The framework we will be building is meant to be a hybrid framework to allow for testing of APIs and UI, and potentially mix the two together for some tests.  We will be using the following tools and languages as part of building, so please follow the instructions for each tool below to ensure you are set up.

If you are using a mac it is strongly encouraged to install homebrew which will allow you to install many of these tools through the command line with a simple command.  For Linux users similar installers such as yum and apt-get are recommended.

Continue reading “Let’s Build a Framework – Day 0”