Database Reset in Automated Testing

By mark.

At TIST conference, after my demonstration, on tester asked me "We saw you running the same test (changing password) twice, before and after refactoring, how could it work second time? The data has been changed from first run.". The answer was database reset, ie. the database is reset to the known state before running each test. As I have been doing this for a number of years now, it almost become second nature to me in terms of automated testing. To make the concept of database reset easy to remember, I call it "The Simpsons Pattern".

Benefits of database reset is so important that you probably will appreciate only after experience it. Once I asked a business analyst to rate his view of importance on database reset in scale of 1 - 10. He answered "9.... No, 10".

  • Develop tests without worrying of data changes caused by tests, what a feeling!
  • Faster test execution. Use seeded data records instead of creating data from scratch every time.
  • Always know data record states, great even for manual inspections of applications.
My typical RWebSpec test scripts like this:
suite "Group of related test cases" do
  include TestHelper

  before(:all) do
    open_browser
  end

  before(:each) do
    reset_database
    login_as("zhimin")
  end

  after(:each) do
    failsafe{ logout }
  end

  test "One test" do
    # might change application records
  end

  test "Another test case" do
    # doesn't matter the above test is run or not, as we start fresh
  end
end

Now comes the hard question, how could you implement database reset as testers? No, it has to been done by developers. If your team is doing Agile (or claiming so), go ask them, and it will be of great help for them as well. From experience, the better architecture, easier to implement database reset and the earlier the better.


Watir Podcast on TestWise

By mark.

My Podcast interview with Željko Filipin (the host of Watir Podcast) and Alister Scott (WebMaster of Watir.com) is now published: "Watir Podcast #34 Zhimin Zhan on TestWise":http://watirpodcast.com/34-zhimin-zhan-on-testwise/.

Ant today is 2nd birthday for Watir Podcast, so Happy Birthday! Thanks Željko and other contributors!

StoryWise is released

By mark.

StoryWise, one of Agileway's Agile Testing Suite product, is a flexible web-based project management tool.

StoryWise is free and open source, released under the terms of the GNU General Public License v2 (GPL).

  • Requirement Management
  • Defect Tracking
  • Project Wiki
  • Time Tracking
  • Gantt chart and calendar
  • Requirement Traceability

Apart from being free, simple and easy to use, StoryWise distinguishs itself from others with one unique feature: Requirement Traceability, a term nowdays gradually fades out due to high hype with low benefits but lots of maintainence work.

StoryWise is different, it allows Requirement Traceability to be achieved simply and effortlessly. StoryWise was first demonstrated at 3rd Tianjin International Conference on Software Testing with very positive feedbacks.

Check it out!

3rd TIST Conference Slides and Videos

By mark.

ANZTB 2010 Conference Slides and Videos

By mark.

ANZTB 2010 Conference

By mark.
I will be delivering a presentation "Testers and Programmers Collaboration in Automated UI Testing" at upcoming ANZTB 2010 conference, see you there! !http://www.iceaustralia.com/anztb2010/pix/gui/pix_splash.jpg!

AJAX testing in iTest2

By mark.
Web 2.0 sites such as GMail grow rapidly, which brings big challenges to web test automation. Different from traditional web sites with "synchronous request/response model", new web sites handle requests asynchronously using AJAX. For example, in GMail, clicking 'Send' button does not land you on 'Email Sent' page, instead you see a '_Sending..._' banner appears and replaced with '_Email sent successfully_' later. The difficulty of testing AJAX web sites are not knowing when the request is processed and verify dynamic updated page contents. The approach used iTest2 is simple, after AJAX call, keep trying next operation with specific time. During that time frame, operation error will be ignored (and try again after interval),
    button(:value,"Transfer").click
    # try up to 10 seconds, waiting receipt date 'showing up'
    try(10) { label_with_id("date").should == today }

    # try up to 60 seconds every 10 seconds.
    try (60, 10) { click_button('Next') }
To assert specific HTML element is shown or hidden
    assert_hidden(:div, "info")  # 
...
is hidden assert_visible(:div, "info")
To assert specific HTMl element is added or removed in a web page
    assert_exists("label", "email_sent") # 

False positive virus alert on TestWise.exe

By mark.

We use free "Bat to Exe Converter"" to create an exe file (with icon) to launch TestWise. Unfortunately, a few Anti-Virus software reported false positive on it, please be assured that TestWise is virus free. Virus Total, an online virus scanning service, reports :

"5 out of 41" Ant-Virus software detect TestWise positive, they are Comodo, Jiangmin, McAfee+Artemis, McAfee-GW-Edition and TrendMicro.

If you happen to have the annoyance, you can try to start TestWise with batch file included.

If you are interested in why exe file generated by "Bat to Exe Converter" may cause false virus alert, please read this.

Load testing with RWebSpec

By mark.
This is one of most asked questions on iTest: "Can it do load testing?". As more projects getting to acknowledge "Automated UI (functional) testing is not possible":http://blog.objectmentor.com/articles/2009/09/29/ruining-your-test-automation-strategy, Load testing gradually is becoming the replacement of 'Automated Testing', which is not right of course. Users of iTest will disagree this permissive view of functional testing, but that's not the topic today. Back to the question, our answer is "iTest is a functional test tool, but the functional test scripts created may be used for load testing, and work has started on creating a load testing IDE". What's special about it?
  1. Reuse functional test scripts

    Different from most current load testing scripts (which work on HTTP request levels), our loading test scripts use same syntax of functional test scripts: RWebSpec, which is known to be very readable. Furthermore, you can reuse the functional test scripts for load tests with only slight modifications.

  2. See load test scripts in action in Firefox browser

    Although execution of load tests is in headless mode, ie, seeing test running in a browser during development greatly improve the experience and confidence. Load tests with RWebSpec just allow you that.

  3. Can verify page contents just as functional testing.
The IDE is still under development, but you can try load tests using RWebSpec now (from command line). You need Java 1.6 installed first, see ReadMe.txt included 1. Download iLoad2 v0.1 here 2. Unzip to C:\ to make application folder C:\iLoad2-01 3. Start a command window, enter the following command to add application to PATH.
C:\>cd iLoad2-0.1
C:\iLoad2-0.1>SET PATH=C:\iLoad2-0.1\bin;%PATH%
4. Preview load testing run in Firefox browser ("install JSSH plugin":http://TestWise.com/documentation/firefox-support first)
spec -fs samples\agile_travel\deny_access_preview_rwebspec.rb
5. Run load tests with a number of virtual users.
spec -fs samples\agile_travel\deny_access_rwebspec.rb
The output shows how long does it take for each virtual user:
  Thread[1] 6.861s
  Thread[3] 7.455s
  Thread[2] 7.674s
  Thread[4] 9.456s
Now let's example the actual load test scripts:
run_with_virtual_users(4) {
  browser = open_browser("http://travel.agileway.net", {:resynchronize : false})
  home_page = HomePage.new(browser) # We cam reuse page objects!
  home_page.click_sign_in
  assert browser.text.include?("You did not provide any details for authentication.")
}

Verify database records in functional testing

By mark.
Function testing, in a certain way, is verifying the data (retrieved or stored at the database) through the user interface layer. Sometimes, it might be easier to verify database records directly, some SQL skills are required though. RWebSpec provides an easy way to do so, in a easy 3 steps. # Connect to the database # Choose the table # Use easy syntax or SQL to do verification
 story "Check Mysql database" do
    connect_to_database mysql_db(:host : "localhost", :database : "lavabuild_local", :user : "root", :password : ""), true
    load_table("projects") # now can use Project.xxx for easy-to-user datbase query on this table

    Project.count.should == 1  # record count
    Project.first.name.should == "BuildMonitor"
    Project.last.name.should == "BuildMonitor"
    Project.exists?(:name : "BuildMonitor")
  end
Check out the _verify_database_spec.rb_ in demo project.