Let’s Build a Framework – Day 3


Introducing the API and Self Navigating Page Objects

Well we’ve reached the end of our little journey and in the course of 3 (and a half) sessions have built quite a handy little framework.   This post will sum up what we achieved in the last session where we added a new feature, self navigating page objects, and introduced a basic API test structure so we can run both our UI and API tests from the same framework.

Before we dig too deep like those little guys in that Lord of the Rings movie, lets take a high level look.

What is a self navigating page object?

When working in a page object framework you will find frequently you just need to get to a specific page, and you don’t care how you get there.  For instance, if you want to run negative tests on a form, you just need to get to the form, and don’t care the path or data used to reach the form.  This is especially relevant for smoke tests, where we just want to validate a page loaded correctly.

We are going to build a feature in to allow us to say navigate to this page I don’t care how.  We are going to do this by adding a generic navigateToPage method that all our pages will have, and each page will have something called a navigationPath that gives instructions on how to navigate to the page.

What exactly is involved in a API test framework?

For the second part of the day we will be creating a framework to allow us to run straight API tests along with our existing UI tests.  Because testing API’s is more programatic and straightforward our framework will be pretty easy to set up.

However, we will add one special piece of functionality called a detokenizer, that can take a API endpoint and magically give it all of the necessary parameters, and information to be a successful request.  This detokenizer can than be used to make much more impressive tests.

Creating a Self Navigating Page Object

Now that we know what our plan is, let’s get started.  First we will add the functionality to make all of our page objects self navigating.

Defining the Navigation Path

To conceptualize how to get to a page we can think of a set of instructions matching our current test layout.

on.<pageName>().<Action>(<Data>)

So lets say we wanted to skip straight to the ebaySearchResult page, we can visualize this like so:

  • Step one
    • Page: ebayHome
    • Action: open
    • Data: none
  • Step 2
    • Page: ebayHome
    • Action: searchFor
    • Data: ‘orphans’

Almost all of our pages can have a similar list of steps.  So now that we know that we can define these steps inside our page objects through JSON:

Screen Shot 2017-01-16 at 11.56.41 AM.png
Defining A Navigation Path for Our Page Object

It’s that simple.  Notice that we are hardcoding the data for the 2nd step.  This is because the concept behind navigateToPage is that the test writer just wants to get to the page, they do not care how.

However, it is conceivable that in the future you would want to allow people to specify which data is used to reach a page.  This is 100% possible, but outside of the scope of this project I leave it as an exercise for you dear reader.

Creating a Navigate To Page Function

Once we have defined our navigation path for all of our page objects (or the page objects we think should have them) we need to have a function that all pages can use, that can read the path and turn it into actual test actions.

We’ll call this method navigateToPage() and because we want every page to have access we are going to create a new custom command for it.

Note:  We could also add this as a method on our PageObject base class to achieve the same effect.  The reason we don’t is because we don’t know if every page will have a navigation path so making it a command allows some pages to “opt out” of being self-navigating.

First lets see what the code would look like to take one step out of our path and execute it as a test step:

 var page = navigationStep.page;
 var action = navigationStep.action;
 var data = navigationStep.data;
//Same as on.<page>().<action>(<data>)
 return on[page]()[action](data);

It’s just that easy.  We have transformed our basic algorithm (below) into code (above)

on.<Page>().<Action>(<Data>)

But now we have a problem.  The above code works great for a single step.  But in a navigation path has multiple steps we are going to need to execute each step in the proper order.

We can’t use Promise.all like in other places in the framework because that runs all the promises at the same time.

Do not panic, creating the method will be surprisingly easy but we will need to use a new concept in promises: “each”.   Promise.each takes an array of data, and a function that returns a promise based off that data.

Note this is a big difference between Promise.all and Promise.each, Promise.all takes an array of promises, while promise.each takes an array of data and generates promises from it!

Lucky for us our navigation path is an array of steps. So to execute all of our steps:

 return Promise.each(navigationPath, function(navigationStep) {
     var page = navigationStep.page;
     var action = navigationStep.action;
     var data = navigationStep.data;
     return on[page]()[action](data);
 });

Finally we throw it all into a single command and we end up with:

Screen Shot 2017-01-16 at 12.13.13 PM.png
Completed navigateToPage command

Seeing navigateToPage In Action

Now that we’ve built it, lets take it for a test drive.  You may remember this test we created on earlier:

it('Should open ebay and search for orphans', function() {

     return on.ebayHome().open()
     .then(function(result) {
          return on.ebayHome().waitForPageToLoad();
     })
     .then(function(result) {
          return on.ebayHome().searchFor('orphans');
     })

...

})

 

With our new navigate To Page we can shorten it to:

 it('Go to eBay Search Results page with navigation path', function() {
     return commands.navigateToPage(on, 'ebayCart');
 });

Pretty amazing right?

Creating The Easiest Smoke Test Ever

Now we have self navigating pages, what better way to show the power, than by creating a smoke test where all we have to do is give a page name, and our test can navigate to the page and validate all elements are loaded.

To do this we can also take advantage of data driving in mocha.   To start lets define a list of pages we want to test in an array.

var tests = [
{
page: ‘ebayHome’,
description: ‘Home Page of Ebay’
},
{
page: ‘ebaySearchResults’,
description: ‘Search Results Page of Ebay’
},
{
page: ‘ebayCart’,
description: ‘Shopping Cart Page of ebay’
}
];

Notice, we added a 2nd field called “description” this field is going to help our test output be more human readable.

A nice feature of mocha, is you can dynamically generate “it” statements by using a basic loop.  So we can take our array of tests and using a forEach loop generate a new test for each page.

 

tests.forEach(function(test) {
    it('Make sure page ' + test.description + ' loads correctly', function() {


   });

});

This will generate 3 tests from our array, one  for each page.  It will also use our “description” so our test output will be much cleaner.

Finally we all we have to do is

  1. Call navigateToPage for our page
  2. Call waitForPageToLoad after we’ve navigated
Screen Shot 2017-01-16 at 12.25.50 PM.png
That’s a Pretty Boss Smoke Test Right There

That might not seem like much but we have just created a way to validate every single page in our application.  Simply by adding an entry to our “tests” array.  We could even go one step further and dynamically grab all pages from our “pages” directory and do the same.  Meaning whenever a new page is added it is automatically part of the smoke test.

Pretty boss indeed … hoss.

Sorry.

Creating an Automated API test framework

Now on to the final piece of our automation framework.  We are going to add a little API testing framework to allow us to run both API and UI tests.  For the sake of brevity we are going to assume you are working with REST APIs and have a general understanding of what that means.

Note: The world of API testing is a big one, and this framework we are going to create is by no means comprehensive it is just a quick and simple one to allow you to get started and get additional coverage.

Starting Our Test Server

In order to test our APIs in a consistent way for all readers, this repo comes packaged with a test server that exposes 2 APIs we can use in testing.  To start the server use the below command from the main directory in the repo in a separate command prompt or terminal window.  Once started leave it running

node test_server/server.js

Our Test Endpoints

Below is a list of the endpoints exposed by our test server that we will be writing tests against

  • POST – /users
    • Takes a request body for a new user to be created
      • Request Body requires fields firstName, username, password, and ssn
  • GET – /users/:user_id
    • Returns a user that has the user_id passed in or returns a 404 if none are found

Creating a Basic Test Using Supertest

What we need to know

To get started first we are going to create a very simple test using the supertest-as-promised library in node.  Mainly because we are already working with promises in our other framework.  If you can’t stand promises there is a equivalent ‘supertest’ library that uses callbacks that you can research and use.

To execute an API we will always need these pieces of information:

  • verb
    • e.g. GET, PUT, POST, DELETE
  •  domain
  • path
    • e.g. /users/:userId/items
  • headers
    • e.g {ContentType: application/json, Accept: application/json }

Additionally for some APIs that use PUT or POST verbs we also need a request body.

Our first test will be to hit our POST /users endpoint without passing in a request body.   This should be an invalid request and we would expect to receive a response with a status code of 400.

Creating a Url Factory

To save ourselves a lot of code duplication firs thing we’ll want is to create a simple url_factory.js library and put it in our /api-framework/lib directory.  The url factory will be where we keep all the domains and paths we might want to hit in our tests.  It will do this by exposing methods.  These methods can take arguments so we can dynamically create paths or switch domain per environment.

First thing we want is a way to get our domain:

module.exports = {

    getApiDomain: function() {
        return 'http://localhost:3000';
    }

}

Because we are using our test server our domain is always going to be http://localhost:3000 but this could change if we need to support multiple environments so you can imagine it taking a env variable and returning a different domain per environment.

Next we’ll want methods that will get us the path for our 2 endpoints.

getCreateUserPath: function() {
    return '/users';
 },

getUserPath: function(userId) {
    return '/users/' + userId;
 }

Now it may seem silly to create a method just to return ‘/users’ but we always want to think in terms of change.  If we have 100 tests that all use ‘/users’ in the test and one day our company changes the path to ‘/user’ without the ‘s’ we now have to update 100 tests.  Whereas with the factory we only need to update this method.

For the get users path we can now support passing in the userId and dynamically creating the path with the userId provided.

Creating A Basic Test

Now that we have a url factory we are ready to create our basic test with supertest.  Supertest is a library that uses a “builder” pattern and promises to execute a rest request.  The standard format of a supertest API call looks like this:

request(<domain>)
.<verb>(<path>)
.set(<headers>)
.send(<requestBody>)
.then(function(response) {
   //Do your validation or other calls in here
});

In this case both headers and requestBody are optional.

Given this we now can create our test.  We’ll create a new test file in /api-framework/tests directory and call it api_tests.js.  We’ll need to require supertest, and for readability we will require it and store it in a variable called “request”

var request = require('supertest-as-promised');

Then we’ll create a new test:

describe('My first API Test', function() {

   //Basic API test for a get request with Supertest
   it('Should return a 400 when I make a request for a user without a request body', function() {
   });
});

It can be helpful when working with APIs for the first time to just write the test using empty variables to make sure we have the structure right.  So lets do that using the supertest-as-promised structure discussed above

var domain;
var path;
var headers;
var requestBody;

return request(domain)
.post(path)
.set(headers)
.send(requestBody)
.then(function(response) {
    //validate here
});

 

That’s our test.  Now all we have to do is assign values to our variables.

  • Domain – We can use the url factory
    • domain = urlFactory.getApiDomain()
  • Path – We can use the url factory
    • path = urlFactory.getCreateUserPath
  • RequestBody – Our test is to test what happens when we send an empty one so:
    • requestBody = {};
  • Headers – We can hardcode for now
    • { ContentType: application/json, Accept: application/json}

Now our test looks like:

var domain = urlFactory.getApiDomain();
var path = urlFactory.getCreateUserPath();
var headers = {
   'Content-Type': 'application/json',
   'Accept': 'application/json'
};
var requestBody = {};

return request(domain)
.post(path)
.set(headers)
.send(requestBody)
.then(function(response) {
 });

So you may we wondering why we are hardcoding the headers.  That is a very astute point.  If we had more time I would create either a common library or a headerFactory and have a method like getJsonHeaders().  But for now we are just hardcoding because they will be the same for every test.

Now we have the “execution” portion of our test knocked out.  The only thing remaining is a “validation”.  For this we are going to use an assertion library for Node/Mocha called “chai”.

Using Mocha Chai Assertions

“chai” is a library built to make english like validations.  To use it we need to add a require statement at the top of our test:

var expect = require('chai').expect;

This gives us access to the full “expect” library.  Once we have this we can do very pretty assertions like:

  • expect(<valueFunctionOrObject>).to.eql(<expectedValue>)
  • expect(<valueFunctionOrObject>).to.not.exist;
  • expect(<array>).to.include(‘<valueInArray>’)

With these pulled in we can add 2 validations to our test.  First we want to validate the statusCode is a 400.  Second we want to validate the error message returned meets our expectations:

 expect(response.statusCode).to.eql(400, 'Didnt get the status code I wanted with this url ' + domain + path + ' : instead got this body ' + util.inspect(response.body));
 expect(response.body.message).to.include('Could not create user because of error: Cannot create user without the following fields');

Understanding the Response Object

Looking at our validations we are referencing properties in the response object.  Let’s take a quick aside to understand what is included in the response, because it will dictate what we can validate:

  • response.statusCode – this is the status code returned by the api request
  • response.headers – these are the headers returned by the api request
  • response.body – this is the JSON body returned by the request (if it is not a JSON request do not use this)
  • response.text – this is the raw text returned by a request (for plaintext or xml requests use this)

The convenient thing here is that we can quickly parse any request body thats Json incredibly quickly.  In our case we expect a response body like

{
    message: 'whoa something went wrong'
}

So we can get the message simply through

response.body.message

The Finished Product

screen-shot-2017-01-16-at-1-35-27-pm
Basic Test Using SuperTest

Creating a More Dynamic Way to Test APIs

The above test is simple enough and you can use this format to write most of your functional tests without much hassle and they will be consistent and maintainable.

But let’s go one step further and just like our self-navigating page objects create a way to take any verb/url combo and have it create a ready to run test complete with valid information.

For example, to test the /users/:user_id endpoint we know we’ll need a valid user_id for successful tests.  To get a valid user_id we can imagine we’ll probably have to create a new user first, if we don’t create a new one we run the risk of having stale data or missing data if someone deletes our user.

You can probably imagine a much larger set of APIs where you need all sorts of data, the end result could be a very clunky and hard to read test.

Enter the detokenizer.

Creating a detokenizer

The concept behind a detokenizer is it can find any “token” passed in, and replace it with a valid value.  In this case we are going to consider a token to be anything like {user_id}.  So anything enclosed in ‘{‘ and ‘}’.

So if  I pass in a string of /users/{user_id} it would return /users/1234 or a valid user id.

The logic for this is pretty straightforward

  1. For Each token found in the string
    1. Go to a special library and find a method called "get_default_<token_name>"
    2. Call that method and replace the token in the string with the value returned

So what’s this code look like:

//Find all tokens in a string and replace them with valid values
 detokenizeString: function(tokenizedString) {
  return new Promise(function(resolve, reject) {
    //We clone the string to make sure we are returning a new string and not impacting the one passed in
    var newStr = _.clone(tokenizedString);
    var promiseArray = [];

   //match all tokens in the string
    var matches = module.exports.findTokens(newStr)
    if (!matches) {
       return resolve(newStr);
    }
    promiseArray = module.exports.callDetokenizerMethods(matches);

   //wait for all our detokenize calls to complete
    return Promise.all(promiseArray)
    .then(function(results) {

      results.forEach(function(res) {
         //results come back in the form {token: '{tokenName}', value: 'value to use'}
          newStr = newStr.replace(res.token, res.value);
      });
      return resolve(newStr);
    });
   });
 },

Now thats a lot of code.  You may notice that its calling some other functions too.  Below is the code for the other methods as well.

callDetokenizerMethods, findTokens and computeMethodName

  • callDetokenizerMethods
    • Checks if there is a method called “get_default_<tokenName>” if so call that method and push it on the return array.  If not reject and complete to inform tester a method is needed.
  • findTokens
    • Uses a regular expression to find all tokens in a string
  • computeMethodName
    • take a token and return the expected method name
//for all the found tokens call the appropriate 'get_default_' method to get a valid value
 callDetokenizerMethods: function(matches) {
    var promiseArray = [];
    matches.forEach(function(token) {
       //Compute the detokenizer method name
       var expectedMethodName = module.exports.computeMethodName(token);
       if (typeof(detokenizerMethods[expectedMethodName]) !== 'function') {
         return reject(new Error('COULD NOT FIND METHOD FOR TOKEN ' + expectedMethodName));
       }
       promiseArray.push(detokenizerMethods[expectedMethodName]());
     });

    return promiseArray;
 },

//Uses a simple Regular Expression to find all tokens in a given string
 findTokens: function(tokenizedString) {
    reg = new RegExp(/{(.*?)}/);
    var matches = tokenizedString.match(/{(.*?)}/g);
    return matches;
 },
//Determines the correct method name to call to get a value for the token
 computeMethodName: function(token) {
    var expectedMethodName = 'get_default_' + token;
    expectedMethodName = expectedMethodName.replace('{', '');
    expectedMethodName = expectedMethodName.replace('}', '');
    return expectedMethodName;
 },

Now in API testing its not just the URL that can have tokens.  Consider our POST api.  For it to be successful it needs a request body like:

{
   firstName: '{validFirstName}',
   username: '{validUsername}',
   password: '{validPassword}',
   ssn: '{valid SSN}'
}

Some of these can probably be hard coded like firstName, but some may need to be dynamic like “user_id” in the above example.  The good news is we can use the same logic (with a little fancy twist) to detokenize a request body.

  1. For Each key in the request body object
    1. If the key is a string call our detokenizeString method
    2. If the key is another object recurse and call our detokenizeObject method

The fancy “twist” being recursion.  If you are not familiar with recursion don’t worry we will paste the code for both of these functions further below, and they wont need to be updated.

Here is what that code looks like

//Takes an object and checks each field in the object if that field is a string we
 //call detokenize String to replace any tokens with valid values
 //If the field is an object, we recurse and call detokenizeBody again for the sub object
 detokenizeBody: function detokenizeBody(inputBody) {
    var self = this;
    //We use cloneDeep to ensure we return a new object and don't actually change the one passed in
    var body = _.cloneDeep(inputBody);
    var promises = [];
    var totalKeys = Object.keys(body).length;

    //if the object is empty no need to go further
    if (totalKeys === 0) {
       return Promise.resolve(body);
    }

    Object.keys(body).forEach(function iterateKeys(key) {
       var val = body[key];
       var newVal;
       //if its a string call detokenize string to return a new string with all tokens replaced with a valid value
       if (typeof (val) === 'string') {
          promises.push(module.exports.detokenizeString(val)
          .then(function(newVal) {
             body[key] = newVal;
           }));
       //if its an object then start over for the new object
       } else if (typeof (val) === 'object') {
          promises.push(module.exports.detokenizeBody(body[key]));
       }
    });
    //once all tokens are replaced send back our fancy new request body 
    return Promise.all(promises)
    .then(function(result) {
       return Promise.resolve(body);
    });
 }

A Quick Apology

Now that is a ton of code posted above, and we don’t have time or space in this already long blog post to dig in and explain each line.  But there are no new concepts in the code we haven’t already covered.  And the code is well commented.   I highly recommend taking a look and seeing if you can figure it out.  But apologize for making you do so.

Creating business specific detokenizer methods

So in the code avalanche above we are trying to find out if there is a method called get_default_<tokenName> in a special library called detokenizer_methods.  This is how we get a valid value.  So for any token we want to cover with the detokenizer, we need to have an equivalent method in our new library.

In our test situation we need a method for user_id, username, and ssn.

These methods are very simple we just need to create whatever data we want and return it in this form { token: ‘{username}’, value: ‘validValue’}.  For username and ssn, we can do this very simply:

//Returns a guaranteed unique user name by using a date stamp
 get_default_username: function() {
    return Promise.resolve({token: '{username}', value: Date.now()});
 },

//Returns a hard coded valid ssn, this might change in the future so we leave the method here
 //Even though this is hardcoded and not asynchronous we still return a promise because the detokenizer treats every method like it might be a promise
 get_default_ssn: function() {
    return Promise.resolve({ token: '{ssn}', value: '755-90-1324'});
 }

Why promises?

We are using promises, because some tokens may require us to actually create new data to get a valid value while others can just use static data or easily created data like Date.now().

Our final token user_id, we are going to need to create a new user.  For this we will need a userHelper library.  This library will create a user using our own endpoints and return that user.  We can then use the userId returned for our valid token.

Here is what our user_helper.js library looks like:

Screen Shot 2017-01-16 at 2.16.52 PM.png
UserHelper createNewUser method

This code is very straightforward, just like our first test we are simply taking 3 arguments and using them to create a new user using our POST /users endpoint.  If the request was successful we return the response body which should have the userId, if not we reject and fail the test.

Now that we have the userHelper we can finish our final detokenizer_method for user_id

 //Creates a new user then returns that users Id to guarantee its always a valid user id for the test
 get_default_user_id: function() {
    //create a new user and return that users id
    return module.exports.get_default_username()
    .then(function(username) {
       return userHelper.createNewUser(username, 'glenn', '123-45-6789');
    })
    .then(function(result) {
       return Promise.resolve({token: '{user_id}', value: result.id});
    })
 },

Now when our detokenizer calls get_default_user_id we go to our helper and create a new user, then we return the Id of that user to use in our test.

Seems like a lot of work to set up.  But from now on we can guarantee any test needing a user_id will have a valid one.  Also, going forward, we can add as many methods as we want for each token making it easier and easier to create tests for new endpoints

Putting it All Together

Let’s look at what our test looks like now using our detokenizer framework:

Screen Shot 2017-01-16 at 2.21.10 PM.png
Testing the GET request with the detokenizer
Screen Shot 2017-01-16 at 2.21.44 PM.png
Testing our POST request with a detokenizer

 

Where To Go From Here?

Now it may not seem like much savings with the detokenizer, but it is the seed from which you can do many great things with API testing.  Imagine you want to any of these tests:

  • use a invalid value for every parameter and test the response
  • use a missing value for every required query parameter or request body value
  • test a missing request body for every POST/PUT request

You can use the concept of detokenizing to test every one of these in a data driven fashion.  In fact the parent company of this post has a blog post covering a framework built exactly on top of this concept.  This framework is responsible for over 4000 tests dynamically generated and automatically detected whenever a new endpoint is added.

A fully formed Detokenizing API Test Framework

At Long Last

Well it’s been a fun 3-5 hour journey creating this framework with you.  Now that we are done, I encourage you to treat this framework not as a finished product but rather the start of your own fantastic framework.  Modify it to suit your company with your page objects and endpoints.  Create new features or change existing features to suit your needs.

It’s been said that great works of art aren’t created, they’re abandoned.  If you’ve followed along with this and created it yourself don’t abandon it now, keep going and see just how far you can get.  Then make sure to let us know.

Thanks to everyone!

 

 

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s