Continuous Testing Server with Functional UI test execution in parallel
Set up Parallel Testing
Distribute tests to multiple build agents to run them in parallel.
BuildWise makes Continuous Testing as simple as it could be (see Screencasts for tutorial). Some succesfully set it all up running by themselves, but we also know some didn't. Because Continuous Testing is the hub of integration, hence requires broad knowledge such as version control, network, Virtual Machine, Shell/batch scripts, test automation, database, ..., etc.
If you have spent more than 2 days on configuring BuildWise for parallel test execution, you probabaly are not doing it right, might be just a simple thing like extra spaces on database password. Seeking professsonal coaching is most cost-effective way.
- Build agent machine
- Install BuildWise agent
- General Settings
- Configure BuildWise agent for project
- Configure BuildWise agent for pre-execution
- Start up BuildWise Agent on boot
In the context of testing web apps, a build agent is considered as a computer with browser, which is driven by test scripts to verify the app.
Build agent machines are be physical or virtual machines, as long as they can communicate to the BuildWise server and the target web sites.
Virtual Machines in computers on LAN
AWS, AZure and Vultr all offer hosting Windows Server 2016 instances. The prices are all quite reasonable, staring about $50 per month.
Physical machines on LAN
Reuse existing physical machines.
Once the machine is set up, install Git and Chrome (and other browsers if necessary).
Verify that you can visit
- BuildWise server
- Your test server (the application under testing)
in your browser.
A BuildWise agent is installable software that runs test jobs, perform the following steps repeatedly:
- Check the server whether there is an active build? If yes,
- Get the test script from the server
- Run the test script
- Return test results to the server.
BuildWise Agent are available on Windows and Mac OS X. As BuildWise Server is 100% open-source (in Ruby), software companies can develop own agents that can work with BuildWise server.
Install BuildWise agent is simple, just download and run the installer, accepting all default options.
Starting with 3 agents
More agents will speed up test executions, but also require more infrastructure maintenance effort.
If you are not sure, we recommend starting with 1 BuildWise Server + 3 Virtual Box VMs in one computer. A typical laptop nowadays with 16GB memory can comfortably run them, and you probably can get it all set up within hours, no need seeking permissions.
Here is an example:
BuildWise Server URL
The URL of the buildwise server, e.g. http://buildwise.server. Click ‘Visit’ link to verify the connection.
The unique agent name (set by license, with exception of Site License) to the Server.
The wait period (in seconds) to check active builds or next tests
Preferred browser for running tests in
The support of this depends on the test scripts. BuildWise Agent will pass an environment variable
BROWSERto test execution, it is up to test scripts to take advantage that.
The directory where BuildWise Agents will save test results (and screenshots in case failures) to. The ‘Archive test results’ tell Agent to retain the history of test results (in timestamped directory name).
Start Agent on Launch checkbox
The agent process (talking to the server) is not started by default, to prevent confusion in case the agent machine is not ready. The user can click the ‘Start’ button under the Dashboard tab to start it.
With this option is checked, BuildWise Agent will automatically starts the agent process on start.
To make an agent to run tests for one project, two settings need to be in place:
The test project checked out (from Git) on the Build agent machine
This only needs to be done once. The directory needs to match the one set in BuildWise Project settings.
In above example, for application WhenWise, its working directory on agents can be
C:/work/whenwise(Windows) or /
It is important that the parent repository of checked out project source on BuildWise Server are the same as the ones on build agents.
Environment variables such as test server URL, …, etc
Special environment variables set set in Agent to pass to test execution. For example, we commonly use BASE_URL to set target server URL, so that Agent 1 -> Server 1, Agent 2 -> Server 2, …, to achive parallelism. Of course, you can set multiple agents with the same URL, this can serve as a form of load testing as well, but the test scripts need to handle side effects (test exeuction need to independent).
Below is an example of use BASE_URL in Ruby scripts, quite simple, right?
def site_url(default = $BASE_URL) ENV['BASE_URL'] || default end #... driver.get(site_url)
Please note that environment variable has to be associated to an application name, so that Agent can support execution of multiple projects. Here is a sample configuration.
Due to the nature of UI test automation, test execution can be suspended, or unepxected siturations (such as browser hangs) can happen. This can affect next build. So BuildWise run a set of pre-test-execution commands, it is configured as below:
You configure your machine to automatically run BuildWise Agent at startup.