January 29, 2011

Estimation, Guesstimation, and Testimation

How long will it take to test?  Sometimes when I hear that question, I cringe.

Don't bother me, I'm Testimating!
(not actually me)

Estimation is the calculated approximation of a result which is usable even if input data may be incomplete or uncertain. [Wikipedia]

Guesstimation is an informal English word derived from guess and estimate, first used by American statisticians in 1934 or 1935. It is defined as an estimate made without using adequate or complete information, or, more strongly, as an estimate arrived at by guesswork or conjecture. [Wikipedia]

Testimation is what you are forced to do when someone asks you to quickly predict how long it will take to test something - something that you've never done before, or seen, or even have any idea about what might actually be involved. [Me] copyrightjoestrazzere

Does this scenario sound familiar?
Product Manager: Hi, Joe.  We're putting together estimates for building a Zerble feature for the Framis product.  How long do you think it would take to test?
Joe: Uhm, what's a Zerble?
Product Manager: Well it's sort of like a Whatsit, only different.  Frankie, the new guy in Development, thinks it will only take him 3 months to finish.
Joe: Are there any written Requirements I could read?
Product Manager:  No.
Joe: Has Frankie designed it yet?
Product Manager: No. 
Joe: When do you need an estimate?
Product Manager: Now. 
Joe: Hmm...

Ok, so I'm exaggerating a bit.

Usually, you get some vague idea about what might be required, with a promise by the Product Manager to be more specific somewhere down the road - you know, after they actually talk with the customer and find out what they need.  And if you are lucky, the Developer may have some concept of the design he will choose - likely something between a quick hack and a rewrite of the entire system - he'll tell you more right after he's finished deciding which technology to use, and after he's finished coding the thing.  And you usually don't have to provide your estimate immediately, end-of-day is usually good enough, and it's only 4:30 now.

What you'd like to do is estimate something like "Somewhere between 1 week and forever."  But, you are a serious professional, and folks expect a thoughtful, scientifically-derived answer from you.  So instead, you answer "If I have to guess right now, I'd say 2 months, but I can't have any confidence in that until I learn more.  I'll give you a more accurate estimate when I understand the Requirements, and have some idea of the Design."

And of course two things inevitably happen:
  1. A formal project plan is created showing 3 months of development effort followed by 1 month of testing.
  2. Someone higher up the food chain instructs the project team to "get creative" about doing better than 4 months total.  Because, after all, they promised the customer that it would be delivered by the end of this month.
I've been trying to think of a good analogy I could use to better explain to folks how inaccurate some of these Testimations must be, and what it feels like to be part of one.  

Imagine you are a professional proofreader.  Your boss asks you for an estimate.
Boss: I'd like you to proofread a book that Ted is writing.  How long do you think it will take?
The last book you proofread was a book of short stories for children.  It took you 4 months to proofread, so you might say "Four months".
Boss: Great!  I'll tell the printers that we'll have the book to them in five months - that should give us plenty of time.
You have a lot of other work to do over the next few months, and you won't be able to enlist any help.  So you mentally go through ways you can shuffle things around, and you agree. 
Boss (one day later): Oh, I forgot to mention, Ted says he's a bit delayed, but he promises to get the first draft of the book to you in five or six more weeks.  Since you estimated four months, and you are a really good proofreader, that won't be a problem, right?
Boss (two days later): Did I mention that the book is about the history of abstract painting in the 20th century?  Yeah, I know you don't know anything about painting, but it's still a book, right?
Boss (seven weeks later): I spoke with Theodopolous, and he's almost finished.  Should be just a few more days after he returns from his vacation in Greece next week.  By the way, he decided it would be a lot quicker if he just wrote it in Greek.  That's not a problem is it?
Boss (four weeks later): Theodopolous said he's running a bit behind, so he decided to get the book to you in two parts.  He'll have his part ready in no more than one month.  His partner,  Frank promised to have his part done at the same time.
Boss (three weeks later): Good news!  Theodopolous has his part - Chapter 1 - done ahead of time!  He'll send it to you tomorrow.  By the way, Francois has decided it will be quicker for him if he just writes Chapters 2 through 36 in French, since he doesn't write Greek or English.  Oh, and it's going to take him a bit longer than expected.  He's estimating four more weeks.
Boss (one month later): The printers said that they absolutely need that book in their shop by the end of this week, since the presses will be busy printing census forms for the following six months.  Ted says that shouldn't be a problem - he's already met his committment.  Let's be creative and get it to them in time, ok? 
Boss (later that day):  I'd like you to proofread a book that Alice is writing. How long do you think it will take? I promised to have it to the printers before they get busy with those census forms next week.
It's pretty clear that the accuracy of a test estimate depends on many factors - some of them out of our control.  With more information, we can be  more accurate.  With more history and practice performing the same task, we can be more accurate.  With more control of the situation, we can be more accurate.  With more flexibility, we can be more accurate.

But with little information, no history, no control, and no flexibility, our test estimate is nothing more than a guesstimate.

I can always give you my Testimation Guesstimation right now.  But if you want a real Testimation Estimation, you might have to wait a little while.

This article originally appeared in my blog: All Things Quality
My name is Joe Strazzere and I'm currently a Director of Quality Assurance.
I like to lead, to test, and occasionally to write about leading and testing.
Find me at http://strazzere.blogspot.com/.

January 20, 2011

Beta Tester Opportunities For Better Testers

A while back I posted about being a Beta Tester

Beta testing can be a nice way to enhance your skills as a tester, and perhaps add to your resume.
Here are a few more opportunities.  As always, be careful out there in Beta land.


The Launchpad team runs a continuous beta program, as well as special beta test campaigns.
You can get access to the beta program by joining the Launchpad beta testers team. There are close to 2,000 beta testers and we encourage enthusiastic users to join that team to get the very latest functionality in Launchpad.


We recently announced the public launch of the new StatCounter site which is currently undergoing beta testing.

Zenph RED

We're extending invitations to beta test RED (working name), Zenph’s innovative new music editing software. RED allows users to edit high-resolution MIDI files for Yamaha Disklavier Pro and Disklavier Pianos. You can read more about RED here. Beginning March 1, 2011, beta testers will be able to download RED through Zenph.com. All we ask in return is that you post your feedback to a forum.


Want to join the Beta Test? I’m looking for active Twitter users who would ideally have a blog to test both the Widget and their Twitoaster profile page. 

This article originally appeared in my blog: All Things Quality
My name is Joe Strazzere and I'm currently a Director of Quality Assurance.
I like to lead, to test, and occasionally to write about leading and testing.
Find me at http://strazzere.blogspot.com/.

January 18, 2011

Book: Glitch - The Hidden Impact of Faulty Software

I happened to be executing a search on Google, when I saw that the company Weblayers was giving away preview copies of a book titled "Glitch". I visited the website and it sounded intriguing, so I requested a copy. Last week, I received a package in the mail containing the book, along with a nice note from the Director of Marketing Communication at Weblayers. In it he said that since the book had already been released, he thought he would send me the full book, rather than just the preview I had requested. Very nice! I'm always happy to be given a book. But if I review the book and write about it, I'll always be honest - good or bad.

Glitch was a quick read, but unfortunately, not really what I had expected. While it did contain some stories of faulty technology in the wild (Visa, Toyota, Varian Medical Systems, EDS), the actual underlying theme of the book is IT Governance.

Automated IT Governance happens to be the focus of Weblayers, Inc. where the author Jeff Papows is President and CEO. So perhaps it's not surprising that IT Governance is heavily emphasized as a major part of the solution to the problem of faulty software, and even cyber terrorism.

Curiously though, the author never actually defines the term "Glitch", nor "IT Governance".

Papows proposes something he calls "the IT Governance Manifesto", and says:
Making this vision a reality will require a cross-section of IT and business professionals, government agencies, and consumer advocacy groups that will join to accomplish the following:
  • Lobby for new legislation that requires more stringent reporting of software glitches in matters of life and death.
  • Impose fines on individuals and organizations responsible for software glitch cover-ups that put consumers' health and/or safety at risk.
  • Require a specified level of IT governance at organizations that produce products that can directly affect a consumer's quality of life.
The author has little to say about QA and testing. He does suggest that "time should be built into the product development and release schedule to include this critical [testing] step". But then he proposes that companies should "Give the developers incentives by offering rewards for those who have the highest percentage of clean code" and "Offer testing teams rewards for the highest percentage of discovered bugs". I've worked at companies who rewarded clean builds from developers or high bug counts from testers - it wasn't a pretty sight with lots of gaming the system, infighting, resentment, and so on!

Overall, I'm not sure to whom I'd recommend Glitch. The back cover says that it was "written for senior decision makers". I would hope that those senior decision makers who decide to read this book would at least talk with their hands-on IT folks - the developers and testers - before they decide to take a deep plunge into IT Governance. I suspect they could learn a lot from such a conversation that isn't covered in Glitch.

This article originally appeared in my blog: All Things Quality
My name is Joe Strazzere and I'm currently a Director of Quality Assurance.
I like to lead, to test, and occasionally to write about leading and testing.
Find me at http://strazzere.blogspot.com/.

January 17, 2011

A New Hobby - Snowshoes


In our continuing quest to get out of the house more, do more things, and try to stay more fit, my wife and I added a new hobby - we bought ourselves some snowshoes.

As with our kayaks, she went for a "girlie" color - powder blue, and of course I got the far more "manly" red.

This past weekend, we had plenty of snow on the ground, the weather was sunny, not too cold, and almost no wind, so we went outside to try them.

On Saturday, we trekked the suburban wilderness of our back yard and surrounding woods for a few hours. Not bad, once we got used to them. Our neighbors looked at us oddly as we walked on top of the 5-6 foot high mounds of new snow in front of the house, but it's not the first time they've gotten a chuckle from us.

On Sunday, we drove to nearby Lexington, and walked (snowshoed?) along part of the Lexington Minuteman Trail. A few others were there as well - either snowshoeing or cross-country skiing. The trail has some nice scenery, and a few interesting markers along the way. And we didn't have to break our own trail unless we wanted to; when we got tired, we could just follow the trail of others for an easier walk.

It was a lot of fun, and pretty good exercise, too. Seems like a nice new weekend activity when the winter weather isn't too forbidding.

This article originally appeared in my blog: All Things Quality
My name is Joe Strazzere and I'm currently a Director of Quality Assurance.
I like to lead, to test, and occasionally to write about leading and testing.
Find me at http://strazzere.blogspot.com/.