When Testing isn’t Testing

Let me ask you if this sounds familiar. You deploy a feature or a site to your customer’s staging server and send them an email letting them know that they can begin testing. After a day they email back that “Everything looks great!” and you deploy the feature to production. A day after that, your inbox is flooded with frustrated emails from your client how “everything’s broken and nothing works.”

What happened? Didn’t the client just say everything was great?

Yes they did, but unknown to you, their idea of testing was vastly different than your idea of testing.

Here was your customer’s “test”: One person in the office opened up their browser, checked to see if the pages matched the comps from their graphic designer, maybe edited a few fields and clicked “save.” They did that once, and they didn’t check edge cases, different browsers, or go through all the features you discussed in the requirements document. Then they sent you an email, “everything looks great.”

Now they’re getting angry emails from their users about how it won’t let them save in IE 6, or Firefox, or how the print margin on the printer-friendly view are off, and they’re asking “didn’t you even test this stuff.”

See the problem? When you told them that there was a “testing” phase to the project they assumed that you did all the technical testing, all they had to do was do a final sign off that everything was in the right place and looked the way it should.

Testing is an overloaded term, what you consider testing (finding bugs, checking if features work the way they’re supposed to, checking different browsers or operating systems) is different than what your customers consider testing. Your version of testing is Functional, while their idea of testing is Verification. A customer considers testing the same way they see approving a print ad or a piece or a design. If what I see on the screen matches what I expect, then it works.

They have that expectation that because they assume it’s your job (the developer or implementer of the project) to do the functional testing while it’s their job to verify.

As with most things in freelancing, this can be solved by properly setting the expectations with the client in advance. You have to make it clear that the testing phase of the project includes functional testing by the client as well.

Test Plans are great for setting this expectation, you don’t have to make them overly detailed but include the following for each test.

  • A number to identify the test in emails/bug reports
  • Steps on how to perform the test
  • Edge cases for each test (like trying to save blank forms)
  • The expected result of each test

You can work with your client to define the tests, but remember you’ll have to lead the charge on this (after all, you’re the technical expert).

I like using Test Plans for large, complicated projects and big feature releases, especially with new clients. They may seem like a pain to set up at first but believe me, the hour it will take to create a test plan will save you and your client a lot of pain and suffering post-production.

PS – Oh be sure to factor creating the test plan and managing the testing phase in your quote. But you know that 😉