Let’s Build a Framework – Day 0

Building An Automated UI and API test framework

A big hello to anyone from the West Denver Test Engineering meetup who came to this page in preparation of attending or watching the 3 part discussion on building an Automated Testing framework for API and UI testing.  And to anyone who has just stumbled across this page a big hello to you as well.

For this blog post we will focus mainly on getting set up prior to the meetup or prior to watching the videos to build your own framework.  We will talk about what tools we will be using, and what you need to install.  In preparation for this I have created everything locally to ensure it works, but have done so on a Mac.  We will include information on how to install and run on a Windows machine as well, however these instructions have not been tested as of yet.  As we get feedback I will endeavor to keep this up to date with the most recent information on tools and installation for both Windows and Mac.

If you’re a Linux user, then you are probably used to trudging up the hard road so good luck to you.  Actually, in all seriousness the steps for the mac should be very similar for Linux users so try to follow those.

The Tools We Will Need

The framework we will be building is meant to be a hybrid framework to allow for testing of APIs and UI, and potentially mix the two together for some tests.  We will be using the following tools and languages as part of building, so please follow the instructions for each tool below to ensure you are set up.

If you are using a mac it is strongly encouraged to install homebrew which will allow you to install many of these tools through the command line with a simple command.  For Linux users similar installers such as yum and apt-get are recommended.

Link to Install HomeBrew for Mac Users



We will be managing the source code for the framework through git.  Specifically we will be using a public GitHub repository.  For each day through the framework we will also store a separate branch in our Git repository so you can go back and forth and see what was built each day.

  1. You can check if you already have Git installed by going to a command prompt or terminal and typing this command “git –version” if you get any output other than command not found, then congratulations you are already set up.


Follow the below link to install git for windows.  This will probably also install Git Bash, which is a bash terminal emulated inside of windows which is helpful for some of the commands we will use:



Follow the below link to install Git on a mac


Node & NPM

We will be writing the framework using Node.js alongside its package manager NPM.  Both of these are usually installed together.  To check if you already have node installed use the below command in a command/terminal prompt.

node –version

Installing Node & NPM on a windows or mac machine

Follow the below link and choose the appropriate installer for the windows/mac machine you are using.  Follow the instructions to install.



While we will not be coding in Java we will need java to run the selenium server.  It is very likely you already have Java installed but to check run the below command in your terminal/command prompt.

java -version

Windows & Mac & Linux

The below link should have information for all 3 operating systems to install the Java JDK on your machine.


Mocha is a test harness we will use to run our actual tests once we’ve written them.  To ensure fewer issues we are recommending that mocha be installed through NPM.  It will also be installed locally as part of the set up section below, but please follow this step as well after you’ve installed Node & NPM.

In a command/terminal prompt run this command:

npm install -g mocha

This should work, for windows users you might have to run this command from a Git Bash command prompt.  But should try from a standard command prompt first regardless.


In building out the UI portion of the framework we will be working with Firefox and Chrome, Firefox support is native to webdriver.io but frequently has issues when a new version of Firefox is out.  Chrome works better but requires the user to have the chromedriver installed.

You can install the chromedriver through NPM like mocha with this command.

npm install -g chromedriver

Note: if this command fails on a mac/linux you may have to add sudo before it

Once installed check that you can reach it from the command line with this command.

chromedriver -v

Selecting an IDE

The final and arguably most important thing you’ll need to create a framework is an IDE in which to do your coding.  There are numerous IDEs available for programming in JavaScript/Node all with various advantages.  I’ve listed them out below.  For the demonstration I will be using Atom, but only because I am most familiar with it, IntelliJ and Visual Studio Code are arguably better IDEs with more features.  But try them out and choose whichever you find most intuitive.


Webstorm is IntelliJs JavaScript/Node offering, and it has the deepest feature set and is probably the best choice.  The catch of course is that you need a license.   You can download a 30 day trial to see if its worth your while.


Visual Studio Code

If you are a C# developer or familiar with Microsoft’s Visual studio, then visual studio code will be a great choice.  It has built in Git integration and a smart intellisense code completion/hint feature.  It’s also free, so there’s that.



Atom is an all-purpose IDE with plugins to support any language.  It’s open source and if you install the right plugins can be very helpful.  The downside is you have to install and configure everything to get it the way you want it.


If you choose Atom I would recommend installing these plugins (called packages) through the settings->install menu.

  • language-js-specs – Will turn on syntax highlighting for JavaScript
  • jshint – Will turn on identification of coding errors

Getting Set Up

That wasn’t so bad was it?  Now that you have all the tools necessary installed lets download and install the basic project that we will be creating our framework in.

Inside your terminal (mac) or git bash (windows) prompt decide on a good directory you wish to store all your git projects in.  I recommend your Home directory.  Then create a new directory called “git”.

Mac and Windows (using git bash prompt)

  1. Go to your home directory
    1. “cd ~”
  2. Create a new directory called git
    1. “mkdir git”
  3. Navigate to the new directory
    1. “cd git”
  4. Now clone the meetup project:
    1. “git clone https://github.com/mustarddemon/westdenvermeetupfw.git”
  5. Navigate into the repository created from the previous command:
    1. “cd westdenvermeetupfw/”

It’s that simple.  Now all we have to do is install it

Getting Installed

Once you have cloned the repository down installing is a breeze.  Just run the below command from inside the repository and it will install everything else you need.

npm install

Getting Updated

Throughout the building of the framework we may add changes to certain branches, so if you are revisiting this after a delay consider pulling the latest changes before going further.

Updating Day 0 command

git fetch

git pull origin master

Updating Other Days

git fetch

git checkout <branch for day you want>

git pull origin <branch for day you want>

Checking If It Works

Now that you are all installed, we are going to run through a series of quick checks to make sure everything works as expected.

Making Sure You Can Start the Selenium Server

From the main repository directory run this command.  If set up correctly this command will run until you stop it with “ctrl-c”.

Windows Command


Mac/Linux Command

java -jar drivers/selenium-server-standalone.jar

If you successfully start the server, leave it running for the next steps and instead open a new terminal/prompt for the following steps.

Start The Test API Server

In order to develop tests for APIs we have built a simple test server to allow us to hit our own apis.  So follow the below steps to make sure you can start the test server and access it.

  1. Run the below command.  Like the selenium server if successful it will continue to run until you close it with ctrl+c.
    1. node test_server/server.js
  2. Once the server is started open a web browser of your choice and go to this url:
    1. http://localhost:3000
    2. You should see a simple message displayed indicating the server was set up correctly.
  3. Once you’ve verified the server you can cancel the command from step 1 with ctrl+c as we don’t need it running for any further steps.

Run A Very Simple UI Test

This is the final step.  Run the below command to kick off 2 very simple tests.  The first should open a Firefox browser and go to google.  The 2nd should open a Firefox browser and do a search for New Camera on eBay.  If both of these happen you are 100% ready to build your own framework!

If you have trouble with this section, try looking at the troubleshooting section at the bottom of the page for potential solutions.

Test Command

mocha ui-fw-tests-prebuilt/basic_test.js

Test Command For Chrome

  • Windows
    • set BROWSER=chrome
    • mocha ui-fw-tests-prebuilt/basic_test.js
  • Mac/Linux
    • BROWSER=chrome mocha ui-fw-tests-prebuilt/basic_test.js

TroubleShooting and FAQ

We will update this section as we get feedback from people trying to follow these steps.  If you have any issues feel free to reach out to me through the West Denver Test Engineering Meetup discussion board and I will help to the best of my ability.

West Denver Software Test Engineering Meetup

Golden, CO
88 Test Engineers

If you are interested in seeing some of the best software testing practices put into practical application, then this is the Meetup for you. We will be covering a wide range o…

Next Meetup

Let’s Build a Framework – Session 1

Thursday, Sep 29, 2016, 6:00 PM
19 Attending

Check out this Meetup Group →


I run the simple test and no browser opens

The most common explanation for this is that the selenium server is not running.  Double check that your server you started in a previous step is still running.  If it is still running check the output from when you ran a test and you might see additional clues as to whats causing the failure.

I run the simple test and a browser opens but no page is loaded

If the browser opens but nothing else happens and you are using Firefox then there was probably a new version Firefox released.  Selenium and Firefox have a complicated relationship, in that whenever a new FF version is released it tends to not work until a new Selenium Server is released.  There are a few potential solutions to this.

  1. Switch to running your tests in chrome
  2. Downgrade Firefox to a known working version

Run your tests in chrome instead of firefox

You can change which browser you are running by changing the BROWSEr environment variable using the below commands:


set BROWSER=chrome

mocha ui-fw-tests-prebuilt/basic_test.js


BROWSER=chrome mocha ui-fw-tests-prebuilt/basic_test.js

Set Firefox to not auto update and downgrade yours to a few versions back

To turn off auto updating you can usually go paste this url into your Firefox browser: “about:preferences#advanced” and change your settings there.

To downgrade to an older version of Firefox you can go to the below link based on your OS.  If you want to choose exactly which version/OS you want you can use the further below link.

Windows (64 bit) Firefox 46.01


Windows (32 bit) Firefox 46.01


Mac Firefox 46.01


Linux (64 bit) Firefox 46.01



Retesting Vs. Regression Testing | Software testing Blog.

A great article on the two different types of testing that people often get confused with each other.

How is re-testing different from regression testing? Is there a difference at all? Is regression testing a subset of re-testing? Quite a few times, testing teams use these two terms interchangeably. However, there is a vast difference between these two types of testing. Let us have a look – Regression Testing Re-testing Definition Regression testing…

Source: Retesting Vs. Regression Testing | Software testing Blog.

Testing vs QA

I wrote the article on Medium a while back, and forgot to link it here.

Turns out, it’s getting some traffic today…thought I should go ahead and get it posted. Be sure to check out the comments, as there is a good discussion happening there too.

Testing vs QA

Fostering Growth

One of the biggest frustrations I have run into as a member of an organization is how the leaders grow their employees. In my entire career, I have only had two leaders mentor me in such a way that I felt like I was growing as both an employee, as well as a person. Why there were not more people doing this, I will never understand. If you want to build an organization that is strong, it only makes sense to develop the next level of people below you in order to ensure that this happens consistently. This Harvard Business Review article says it best:

Regardless of what else you expect from your managers, facilitating employee learning and development should be a non-negotiable competency.

So how do you do this? How do you become a manager that people want to work for, and more importantly, how do you develop those people into the next wave of leaders in your organization? The first thing to realize is that there is no magic bullet. There is no secret recipe for success that is 100% repeatable. Instead, there are some key principles that you should keep in mind, and strive to work towards.

People vs Resources

I’m not talking about using resources other than people, but rather how you look at the people you have. Do you treat them like a resource to use as needed, or do you treat them as human beings, and use the individual/unique skills they bring to the table? Having the ability to recognize people for the unique individuals that they are is a lot tougher than it sounds, and takes deliberate and intentional effort. It is far easier to see them as a resource with skills that are defined by a job description, but that does not do anything to further the person, nor does it truly help further the organization. When you just throw resources at a problem, eventually you will run into a scaling problem.

Personal Interactions

When you connect with people on a personal level, they feel valued, and their desire for success, as well as their level engagement go up. In every position of authority I have ever been in, I have gotten the best results out of the people I was able to connect with on a personal level. This not a simple matter, and it is likely that you will have people on your team that you just can’t connect with. This is one of the key reasons why cultural fit is such an important part of the hiring process. When you spend as much time with your coworkers as you do your own family, you naturally want to enjoy that time (and conversely, your employees want the same thing). Having this feeling of camaraderie and friendship will lead the members of your team to want to do better by each other, and will result in a much higher level of efficiency.

Career Development

The resources on your team, regardless of where they are at in their career, need to be treated as a plant would be. They need fed and nurtured on a regular basis in order to keep them healthy and productive. One of the ways I like to do this is by sitting down with each person and outlining where they want to go in their career, and then take that and work with them to put together a list of goals for them to work towards. These goals should be S.M.A.R.T., should be a mixture of individual and teams goals, and should be measured on a quarterly basis at a minimum (I like to review during monthly 1-on–1’s). By including the employee in the process of creating the goals, you enable them to take ownership of their career, and thereby fostering better engagement on their part.

I know a lot of this seems pretty simple and basic on paper, but it is surprisingly difficult to accomplish in reality. The key is to make a deliberate choice with each and every employee to connect with them on their level, and be genuine in your conversations with them. You won’t win them all, but trust me, word will get around, and you’ll find that people are wanting to come to your team or organization. It’s not easy, and there are definitely some bumps along the way, but you will find it all that much more rewarding!

Building for the Future

One of the hardest things in a leadership position is figuring out how you want to build and structure your team. It makes sense then that your hiring process is an important part of this. Identifying talent inside of an organization is hard enough on it’s own, but trying to evaluate a complete stranger from nothing but a resume and a conversation is difficult at best, and usually far tougher than that. So, given that this is such a difficult process, what is the best way to find and identify those people that will move your organization forward?

If you’re me, you start the process by asking about their Tetris experience. I’ve been asking this question for years, and while I have often taken flak (the good-natured kind!), it has thus far proven to be a very effective barometer of a candidate’s critical thinking ability, as well as their ability to quickly problem-solve. From my perspective, these are the most important qualities a person can bring to the table, and are absolutely crucial if you want to build a nimble and agile team that is capable of functioning at a high level.

Another key characteristic that I look for, is their ability to fit in culturally. When I interview someone in person, I am most concerned with how well they connect with me, as well as the rest of the team. For us at HomeAdvisor, a candidates ability to fit in culturally is just as important as their technical abilities, and in some cases, even more important. We are in the midst of transitioning from a startup-type of environment to one where there are multiple delivery teams, and we need to have people that are capable of taking the QA direction and focus and integrate it within their teams.

Finally, the last key thing that I look for in a candidate is their passion and enthusiasm for QA. I want people on my team that are genuinely excited to deliver quality software. I want people that are engaged in the local meetups and communities in the area, and want to broaden their skill set. I want people that want a career in QA, not just a job.

A Change of Course

Over the last 10+ years, my career has taken a number of twists and turns, and my writing has predictably followed those same winding roads. Now that I have entered the next phase of my career, it’s time for my writing to do the same. When I started writing this blog, I was focused primarily on the hands-on aspect of testing, and the tips, tricks and gotcha’s of that side of the equation, but now, I plan to shift the focus and content to be more leadership and career building.

My goal for this blog going forward is to help prepare the leaders that are out there trying to put together the best QA teams they possibly can, and to create a resource for taking your career to the next level.

Thanks to everyone who has stuck around through the long hiatus…hopefully the next few months more than makes up for it!

QA Logo 250

Helpful QA Resources

I was recently asked about some resources to help figure out a good QA strategy, and after digging through all of my links and resources, realized it made a lot more sense to compile it all into a blog post that I can just point people towards. I will add additional content as I come across it, or I think it is relevant.

So without further ado, here are the links!

Discussions/Presentations/Blog Posts

Agile Testing: A Lesson in Diversity

Test Specialists Are Essential Members of Software Teams | A1Q1 Blog

The QA Mindset | Rands In Repose

How to Break the Software Rewrite Cycle

The Forgotten Layer of the Test Automation Pyramid

Guidelines for Automated Testing

Avoiding the most common pitfall of large-scale agile

Shields Down

Continuous Integration & Delivery – Illustrated

Five Ways to Make Test Automation Fail

Why Didn’t You Find That Bug?

Michael Larsen on Testing Career Paths


Technical Approaches/Implementations

Distributed JMeter testing using Docker






List of QA Twitter accounts ADDED 2/24

Slack – TestersIO ADDED 2/24

Slack – #testing ADDED 2/24

5 Signs You Might Be a Workaholic – Forbes

5 Signs You Might Be a Workaholic – Forbes.

Workaholics have a lot of bad habits that can hamper health. Constrained for time, some turn to junk food, some inhale lunch at their desks and others skip meals altogether. Exercise is often abandoned and sleep habits get thrown off schedule.

I hate that it is true, but I definitely suffer from this.  One of my goals for this year is to be a lot more intentional about my time, and this is a great place to start.

The Weirdest Interview Questions Hiring Managers Ask

The Weirdest Interview Questions Hiring Managers Ask | Fast Company | Business + Innovation.

“It’s to test a job candidate’s critical thinking skills, see how they think through a problem out loud, solve through a problem and come up with the best solution, not always the right solution, but the best solution,” Scott Dobroski, Glassdoor’s community expert, told Fast Company.

An older article, but an interesting one.  I’m always interested in how people find and sift through talent, and odd questions always come up as an effective method.  I personally like to ask if the candidate has ever played Tetris, and if so, were they any good at it.  Weird, right?

My Favorites

I created this post as a page initially, but thought I should also go ahead and make it a post as well.  If you want to see the most recent version, you can find it here.

I often get asked questions about what I would recommend for people, or what I use, so I thought I would put this page together to help people find out more details. The format is inspired by folks over at The Setup, who follow it for their excellent Interview series

What hardware do you use?

I currently have a 2013 11″ Macbook Air that is fully loaded with 8gb RAM, and 512gb HD.  This thing runs my life, and I use it every day. My mobile needs are handled by a variety of devices, namely a iPhone 6 Plus, that is my main device when I am away from my laptop. I have it setup so that I can do 90% of my daily tasks right from there if need be. This is especially handy when I am running from meeting to meeting every day. I also have a 1st-gen iPad Air, and a 1st-gen Retina iPad Mini that I alternate between when I want a bigger screen for media consumption. Both are setup similar to my iPhone, but their usage has gone down since the Plus came into my life. I also have a Surface Pro 3 that I use for work purposes, but it pretty much never gets taken off my desk anymore. It’s a good machine, but for the way I work, it is more of an impediment than a help. And lastly, for audio consumption, I have a pair of Beats Studio headphones that I use when at my desk at work, and a pair of Motorola Bluetooth headphones that I use when I am working around the house.

And what software?

I am constantly trying new things out to see if they make life easier, but the apps I list below are the ones that are currently in use on a regular basis.


The main apps that are always open on my latop are OmniFocus, Outlook 2013 for Mac, Spotify, Skype, Airmail, Safari & Ulysses 3. If one of those apps are not open, I’ll know pretty much right away as most of my daily activities happen within them. I also use Lync, Day One, nvAlt, 1Password, Evernote & Transmit on a semi-regular basis, along with a handful of other apps for random things I need to accomplish. As far as utility apps, there are a couple that are always running: Caffeine, Bartender, Dropbox, OneDrive, BackBlaze, Keyboard Maestro, TextExpander & BetterTouchTool. These apps are always there, and always on, and are key to getting through my pile of to-do’s for the day.


IMG_0673I use a LOT of apps on my mobile devices, although the list has consolidated somewhat since iOS 8 has come out. As you can see from the screenshot of my homesceen, there is a lot of variety on there. I also have a send page that contains all of the stuff I use on a regular basis, but maybe not daily. These apps include things like WordPress, Editorial, Screens, various financial apps, cloud storage providers, and other messenging apps. A few notes about a couple of apps on my homescreen: Acompli is simply the best mail client you can get on a mobile device, especially if you use an Exchange server due to it’s native support (Microsoft just bought them, so it’ll just get better); Drafts is where all of my quick notes go; Flipboard is where most of my news comes from; Overcast is my podcast app; and finally Unread is easily my favorite RSS app. Also, I used to use Pocket for my read later service, but went back to Instapaper, and have been very happy since doing so. The experience is far more pleasant, and cleaner overall.

What would be your dream setup?

A retina version of the Macbook Air (which I have heard is coming in the near future) would be my ideal daily computer. Pairing that with a retina 27”-30” display that I can hook up to at my desk, and I would be in heaven. For everything else, I am pretty happy with where I am at. I am constantly trying new things, and looking to simplify the experience, so if you ask me this question next week, it might be different!