Parallel Test Execution
- View error screenshot
- Create To-test-list
- Priority - dynamic ordering
- Auto rerun - reduce false alarms
BuildWise agents take screenshot of the browser when an error occurs, which might help analyze the failure quickly. Click ‘Failure’ link.
Click ‘Screenshot’ tab to view.
When a UI test step starts, it will first lists all the to-be-executed test scripts files.
By default, all test script files under a folder (e.g.
spec) will be included as the regression suite. This is set in Rakefile:
desc "Running tests distributedly" task "ci:ui_tests:full" => ["ci:setup:rspec"] do build_id = buildwise_start_build(:project_name => BUILDWISE_FULL_PROJECT_ID, :working_dir => File.expand_path(File.dirname(__FILE__)), :ui_test_dir => ["spec"], :excluded => excluded_spec_files || , :distributed => true ) #... end
With every test script file (in distributed build), you can see a priority number (shown in last column). This determines the execution order. In other words, tests with higher priority will run before the ones with lower priority.
The priority is auto set by BuildWise based on two factors:
Recent build results of each test script file
The failure of most recent build will get high weighting.
Average execution time
For tests about the same priority, the tests took longer time will get slight higher weighting.
You can also override the default order, by simply clicking the priority number and set a new value.
The execution order changed.
With parallel test execution using BuildWise agents, BuildWise supports auto-rerun, that is, if one test failed on one build agent, server will give it another go by reassigning the test to another build agent. The auto-rerun can be turne on / off in project settings.
Here is a re-run example. The test failed on Agent Win7-BWA-06, then rerun it on Win7-BWA-04 and it still failed.
The below is rerun successful case, which means, we saved a false alarm.