Test Automation vital to Independent Software Vendors

By mark.

Many experienced software programmers's dreams are to be an Independent Software Vendors (ISV), which are evidenced by the best seller book "The 4-Hour Workweek", and "Micro-ISV: From Vision to Reality".

Being an ISV, the resource is limited, most likely only you (and your loyal partner if you are lucky) are to do all, in you spare time (don't ruin your family life). Working harder is not often enough, working smarter is the key.

Months ago, My brother, a dentist and his partners opened a big dental practice. They need a medical practice system, the best quote they got was $25000. My brother told me casually that the software not stable during the trial, but no other choices. I offered: I maybe can write one for you (I never did this kind of system before). I got my first release out within a week (spare time, about 20 working hours) for feedback, and in production within one month. Now they have been using it with satisfaction for 6 months. Just last week, one fellow dentist visited my brother practice and saw the system, he showed great interest and wanted to adopt it.

How? (My brother and I located are in different countries) the secret: test automation. To get their feedback, during the peak time, I released a new version pretty much each night. The automated tests prevent me from making mistakes. Seeing is believing.

  • TestWise test case stats
  • BuildWise CI build report (28 mins with database reset, that's quite a lot of tests)
  • StoryWise requirement coverage
I have two friends who built a very nice web application. During one discussion, they shared their slight concerns on observing more competitions, which is inevitable (Web applications means global competition). I said: "Keep improving your app. Don't worry until one of your competitor discovers TestWise."

Software Test Automation - Public Servants Losing Jobs - Disability Fund

By mark.

These are seemly three totally unrelated topics, if you are not rush, read on.

We all know about 'government waste', it is all tax payers' money in the end. Not many tax payers knowing the scale of waste in bad IT systems, it is worse than pissing against the wall.

To be aboslutely objective, I will just quote the news headlines on some big local IT projects.

Queensland Health Payroll

Gold Coast Payroll

Brisbane City Council Payroll

It is not hard to imagine how much tax payers' money were wasted? Remember, payroll system is a only one kind of system. While Queensland is still in the middle of mining boom, the government is in huge debt. The largest sacking of public servants is about to happen next month: around 20000.

Also as a result, Queensland government is not willing to support National Disability Fund.

As an IT professional, it is heart-breaking to see IT disasters contribute (again) to misery to people lives, particularly disadvantaged ones.

We all heard of 'Test Driven XXX', though many haven't seen one working. Speaking of my experience, software test automation is the simplest, quickest, and most accurate way to measure whether the vendor software is up to the job. Don't fall into those fancy talk and slides, just ask: "Show me how do automate test your application?"

Some might argue it is just your opinion. Check out Auditor-General's report on QLD Health Payroll: Pay system not properly tested. QLD Health payroll is a IBM-SAP project, now a simple question: "How many IBM rational test software sold to QLD government after this report?" An even simpler and logic question: "Are they being used?"

Automate Windows Native Apps with RFormSpec/AutoIT3

By mark.

"Can TestWise test Windows native apps?" I received this question now and then even the person asked knew TestWise is for testing web applications.

My answer is: YES or NO. (I know it is sounding like a politician). First of all, testing Windows native apps is a lot harder, as there were no standard controls. Identifying some controls are particularly challenging or sometimes impossible. Using screen coords is not a good idea, as the test scripts are too delicate. To make it matter worse, assertion is very limited. For that reason, I prefer the term "automation" rather than "test automation" for native apps.

But there is still value doing automation for native apps. The technology I use is AutoIT3, a free and quite widely used windows scripting engine. AutoIT3 comes with Window Info tool, which can be used to identify control IDs or Control Coords

It is important to note that better using Window based Coord Mode, so that if have to control mouse in your test script, the test script still work when window moved.

I like Ruby, so I created a wrapper for AutoIT3. It is called RFormSpec, free and open-source. TestWise comes with RFormSpec, in many ways it works the same way: Page Object, Auto Complete and Refactoring. By using page objects, the automation script is quite maintainable.

For people know me well, I wouldn't talk something without real demo. Here is a Screencast: Automating Native App: HeidiSQL, and you can find a sample test project form_hedisql_ui_tests under samples (from v3.0.5).

Wise Testers

By mark.
At AgileWay, we used the term 'wiser tester' to address our customers.

Wise: "having the power of discerning and judging properly as to what is true or right" - dictionary.com

Just then I watched one introduction video of a testing product, the same old trick: 'scriptless automated testing' and 'record and playback'. I don't understand why on earth still some testers/testing managers/CIOs buy into it. If it is that simple, why your organization not using it now? More insightful thought: if it works as it advertised, list reasons to why your company cannot replace you (as an automated tester) with a new graduate. I have seen too many times: * A new test manager/CIO is ambitious to introduce test automation (it is nothing to brag about manual testing) * Purchased a very expensive testing tool package * Testers (assigned to do test automation) became slaves of recorded test scripts and didn't work * Report: not suitable for the organisation * Keep paying remaining yearly (usually 3) product support You can see obvious flaws there. If the testing product was as good as these salesmen said, why not ask them cutting half sales talk time and write real tests for typical web applications in your organisation? (Testing, compared programming, is practical and knowledge is quite transferable. I usually tell my potential customers, after this brief talk, if permitted, I will spend 10 minutes with your tester writing some real tests for your app.) Now I am getting to understand the same stories repeated again and again in organisation are just games, especially common in governments. This might help you understand why so many governments in debt crisis, :-). If might take efforts for projects to decide suitable programming architecture: Java, .NET and Ruby. But for automated functional testing, in my opinion, it is black and white. Wise people takes the control, wise testers will ask logic questions such as: * If programmers made a change to web pages, how I can maintain the test scripts effectively? * How to handle popups such as user login? * When I have a lot of test scripts, it takes too long to execute, how can I manage better? * How can I include test automation execution in a Continuous Integration process? (Show me an working example) * How easy is your test scripts to read? * ... In developed countries, such as Australia, more and more IT jobs are out-sourced to India and other countries, that's the fact. Just like manufacturing, to compete, you need to improve productivity and innovate (like FaceBook, 100+ users use Watir to do testing, guess what, a free open-source test framework). When I see unwise testers/managers jumped up and down on setting manual testing processes and against agile development, I don't what to say (may be 'sad little creature' in one of Pixar movies), it just seem to me that someone is digging a tomb to bury himself. The quick feedback loop time in agile teams keep your jobs here. Let me tell these unwise testers, the testers in out-sourced countries follow the boring specified manual process better than you.

TestWise Community 3 supports Selenium-WebDriver

By mark.

A couple of days ago, a contract tester, who is approaching to the end of his contract, told me that he registered a QTP course in the hope of helping him with job hunting . I asked him why not learn Selenium. When I showed him the job stats (see below), he was totally surprised: Selenium is currently leading the testing skills in terms of job demand for software testers.

Skills Job Ads (US: dice.com) Job Ads (AUS: seek.com.au)
Selenium 716 127
QTP 612 64
Watir 72 1
Cucumber 130 62

Sources: Dice.com, Seek.com.au, Date: 2012-06-07

Although I didn't have the stats of last year handy to show you, I can assure you that QTP was slightly ahead of Selenium.

Now I am pleased to announce that TestWise Community Edition v3 is available to download, and the highlight of this release is that the community edition now supports Selenium-WebDriver. You have no excuses not to learn Selenium now.

TestWise community edition is free for personal, open-source projects, charity and educational organization. You may use it at work for commercial purpose (i.e., you are getting paid for work using TestWise) under one condition: only one copy of TestWise Community Edition is allowed in a team or company. Basically the principal is: if companies or organizations are getting commercial benefits, they should purchase the Pro edition. Otherwise it is free to use.

TestWise 3 is around the corner and change of pricing

By mark.

We are excited about the coming-soon TestWise 3, some of feature highlights:

The early access version can be downloaded at 'Practical Web Test Automation'.

Along with TestWise 3, there will be also major releases for BuildWise and StoryWise, our free and open-source Continuous Integration Server and Requirement/Release management tool.

Occasionally we receive feedback from testers who love TestWise, but got frustrated: unable to get approval from the management (mostly big companies and government organizations). These managers's reasoning was "if our current testing tool cost tens of thousands didn't' work, how can this one for just $449?"

TestWise's price has not changed for 3 years since its first release (Jan 2009), we now find it necessary to make an adjustment: TestWise 3 will be priced at $1000 for named license, just enough to make it into the thousand-dollar category, which is still a small fraction of other commercial testing tools. Also we will offer one-month standard test automation and CI coaching service (worth $1000) for free, stay tuned for more information about our support and service changes. We believe this move provide better benefits to our customers and help TestWise reaching more testers in big organizations.

Rest assured for about-to-buy and existing TestWise 2 customers. Customers who purchase TestWise 2 before July 1, 2012 (basically next month) will get free upgrade of TestWise 3, existing customers who purchased TestWise after June 30, 2011 also qualify for the free upgrade.

We believe no one shall miss the fun and benefits of test automation for affordability, if you have concerns about this, please fell free to contact us. After July 1, we will still provide TestWise 2 at the current price, but without updates and free upgrades. There is no change to TestWise 2 Community Edition, which is always free for personal use or only one use permitted in a team for commercial use.

BuildWise 0.5 released

By mark.

Recently I received some enquiries and feedback on Continuous Integration (CI). Even though, traditionally, testers have nothing to do with CI except in rare true agile projects. So I am happy to see that proactive and wise testers are pursuing a better working life with CI.

With BuildWise, an easy to use CI server with special catering for automated functional UI testing, testers can get on enjoying the benefits quickly. Here is a new screencast showing setting up BuildWise on Mac in minutes:

  • Install BuildWise (basically unzip and run)
  • Set up a new CI project in BuildWise
  • Trigger a build
  • See CI steps
    1. Check out source
    2. Database migration
    3. Unit testing
    4. Automated UI tests in Chrome Browser
  • View test scripts
  • View test results while test execution in progress
  • Export test report in Excel

There will be more screencasts (e.g. setting up on Windows) on BuildWise on the way, which can be found at Practical Web Test Automation book site.

BuildWise v0.5 is now available for download at the book site, please note windows users need get latest RubyShell v1.9.3 installed first.

Why test processes/documentation mostly are waste of time?

By mark.

Firstly here are some facts:

  • Regression testing is most time consuming
  • Testers generally don't like writing test steps in word-processing tools
  • New generations who get used to YouTube nowadays have less attention span on reading texts.

Logically, a test process shall address to the biggest hurdle test teams face: regression testing. Sadly, I never see one even bother to mention it (and rarely mentioned at software testing conferences).

Let's say, a project follows a process, test plan, test design (in Word), after peer review, test executions, ..., etc. X number of defects were found. Now a new build comes, which includes several change requests of some new or updated functions besides bug fixes. Now as a test manager, do you ask your team just re-test bug fixes or a full regression test? As we all know, programmers do introduce accidental changes. The so-called standard process or documentation won't help you there. (In projects adopting agile or iterative methods, a new build comes out frequently).

My point: without effective and sustainable regression testing, testing processes lack of foundation.

Also consider this:

  • How confident are you that ensure all testers update the test design/test steps to reflect new changes made to the system?
  • How to prevent misunderstandings from reading the same test design document?

What's the solution? My answer is simple: executing tests as a part of Continuous Integration. As Lisa Crispin pointed out It would just be crazy to not do it (CI)". With tests being continuously executed in CI, then your test design/scripts are living things, not just lifeless paper documents.


Verification efforts in total software development cost: more than half!

By mark.

We all know that testing/verification activities are important in any software project, but how important? Most referenced paper on this topic I could find is B. Hailpern and P. Santhanam's paper on IBM Systems Journal in 2002. They stated:
"In a typical commercial development organization, the cost of providing this assurance via appropriate debugging, testing, and verixEFxACx81cation activities can easily range from 50 to 75 percent of the total development cost." - IBM SYSTEMS JOURNAL, VOL 41, NO 1, 2002

This article 'The Case for Automated Software Testing' by Bernie Gauf and Elfriede Dustin, IDT contains further survey statistics on the time spent on testing out of total software development cycle:

Although the numbers vary, it is safe to say mostly will agree to the ratio in the range of 30%-75%. The average: >50%. Nowadays software are released more frequently, i.e, requires more testing effort. If we take this into consideration, the figure is even higher.

Some might find about more than half effort on testing/verification hard to believe, my observation is in align with the authors above, that's is reality in many software projects. Now I let me explain, verification in software department is not only limited to testers (what's why I used 'verification' instead of 'testing'), most of team members in a software projects doing verification:

  • Testers test a build against requirements
  • Programmers verify functions being developed or bug fixes
  • Business Analysts verify the implementation matches customer's expectation.

Please note that they do these activities pretty much every day.

If your software project is late, over budget and looks like a mission impossible, verification is the first area you shall look into for improvement.

Print User Story Cards in StoryWise

By mark.

Agile projects, even fake ones (which don't do automated testing and continuous integration at all), use story cards, that is, writing requirements on cards (commonly, index cards used). But the fact is that nowadays most of people's handwriting, like mine, are not very recognisable (a less polite word: sucks). Story cards are conversation starters in agile projects, but only can be effective if team members can read the text on them.

Almost every project team I am involved likes the idea of printing story cards, this can be easily done in StoryWise. After entered or imported user stories in StoryWise, you can generate story card PDF from user story list

or from the story details page


, and you will get

Then you set up your printer with specified paper size 152x102mm to print it out on index cards.