Ramblings from the Warrior's Den
Wednesday, August 18, 2004
 
Just how well is your software being tested anyway?
Part 1: It runs, ship it!

Recently at work, several people were having a discussion regarding a new test bench being set up for some upcoming localization testing work on the project I'm working on. Assuming I didn't get totally lost in the back-and-forth conversation of exactly what sort of site heirarchy was needed for this effort, my role in all this is to set things up, but the numbers of servers needed, and exactly where these are supposed to come from varied wildly throughout the conversation. At one point, the number of servers required had reached as high as eighteen, before cooler heads (and presumably the available budget) prevailed. Fortunately for me and my my office mate, we managed to convince the test lead that this whole setup probably should be placed in a lab, rather than my office.

In addition to this discussion, This recent post by Chris of the Software Test Engineering @ Microsoft Blog, have led me to give some thought to the subject of exactly how well the software we put out is being tested. I'm probably going to do a series of posts on this subject, assuming I stop being lazy enough to write stuff every once in a while.

If you're familiar with the process of software testing, you know that the testing effort that goes into a typical piece of commercial software is extensive. You also know that there is only so much testing that you can do. Chris touched on this subject in his post:

This is one of the key problems of testing big software like this. You can test and test and test, but it's a bit frustrating to know that even with all your work someone, somewhere will have a problem with your stuff. I had a great conversation with two developers at lunch today about this. Once your software stabilizes testing becomes a low-yield game (tangent: at this point one of the developers cheerfully said “not when you're testing my stuff!”) You test a lot of surface area, and don't find many problems. This is one of the hardest times for testing, the temptation is to say “yep, it's done!”

This is a key dark art for shipping successful software. Everyone wants to be done, but someone has to have a good idea of when done actually happens. Ship too soon and you've got a buggy product out there. Ship too late and diminishing returns has kicked in so your quality isn't much better, but you just lost a lot of money by not being out in the marketplace. Every time software ships someone, or hopefully a group of someone's, thought about these issues an made a trade-off.


While this is true, it's not always quite that simple. Sure, the omnipresent spector of the schedule looms over your head, and you don't want to be the one who has to explain to the VP in charge exactly why you just slipped the ship date by six months, but it's not just a matter of popping out of your office one day and yelling "Ship it!" down the hall either. As a software tester, for the most part you're going to have someone (probably your lead, based on a schedule prepared by a Program Manager) telling you what to do at any given time. The process of getting a product to market is very structured. While this ensures that the basic functionality and features of a given product get tested thoroughly, it does also have the effect of having the testers spend quite a bit of time running through the same tests repeatedly. Depending on the way things are scheduled and the amount of test cases (and how much of it is automated,) some groups opt to run through a test pass on a weekly basis, especially as they approach a major milestone, which generally requires zero active bugs, followed by a period of escrow, in which no major bugs are found, before that particular milestone can be considered complete.

Of course, along the way, bugs are going to be found. Lots of them. Actually, I should probably back up a second here, and point out that in terms of the software development process, "Bug" is something of a generic term. What most people would think of as a bug is generally referred to as a "code defect". The term "Bug" is used to refer to just about any issue with the product that needs attention, be it a code defect, a string that needs to be fixed, a doc issue, s DCR (Design Change Request) or any number of other issues that may crop up. Just because you've filed a bug against the product doesn't mean that it'll be fixed either. Dev resources are lmiited (in fact, a lot of teams have less devs than testers, although I've heard it recommended that for ideal coverage a 1:1 ratio should be maintained.) If you sent all the bugs directly to the devs to fix, you'd be more likely to end up with your dev team barricaded in their office and surrounded by the SWAT team than you would to end up with a quality product. Therefore, you need what's known as a triage to sort things out.

To a lot of testers, the triage is some sort of mysterious process where bugs go in to get chewed up and spit out, often with great force. To be honest, having only recently worked on a team where more than 3 or 4 people attend the triage, I actually think I might prefer that the process remained a bit more mysterious. Getting your bugs tossed back in your face with a big "Postponed" or "Won't Fix" stamp is one thing, having it happen while you're sitting there watching is another. I can now see why it is that it is generally recommended that triage be left to the test leads and the PMs. Of course, even if you are going have half the bugs you file thrown out or punted (especially as you're getting close to shipping,) triage serves a vital purpose. in order to keep things on track, you need to make sure that the major issues get taken care of first, and also that they get to the appropriate people to take care of them. In the end, this is also going to leave a significant number of Won't Fix bugs in the product that didn't meet the criteria for being fixed. These are generally left to the tester's discretion as to whether or not they want to push back on them and try to get a fix. For the most part, these are going to be relatively minor issues anyway, and often have little effect on the end user experience.

Next time: Some discussion about the environments in which software is tested.
Comments:
Hello, you have a great blog here! I'm definitely going to bookmark you!
I have a opt in mlm lead site. It pretty much covers opt in mlm lead related stuff. Check it out if you get time :-)
 
One good resource for computer software and many more free software alternatives is Software4YourSuccess.com
Yes it is my site and I would love for you to drop by for a second. From there you will have free access to several of my products such as The Marketing Toolbar (which is goldmine of information on how to do things quickly and on the cheap, thus saving you time and money).
Also for webmasters I have created Webmaster Wizards, which will help you with almost every aspect of putting code on your site from things such as legal and privacy disclaimers to popunders. I have so much content and free software there I could fill up this whole blog, so check it out Thanks, Sincerely Rob Rudd
 
You have nice blog here! If you are interested, go see my internet marketing toronto related site. It isnt anything special but you may still find something of interest.
 
Nice blog! I have a free lead pot selective solder site I thought you and your visitors might like.

Click on free lead pot selective solder to check it out. free lead pot selective solder
 
hello, your blog is interesting to read, I have a website about internet marketing, it should be helpful to you in making more web money. This is a global trend that more and more people can make a living online, so make sure you visit it.
 
Hey, you have a great blog here! I'm definitely going to bookmark you!

I have a internet marketing strategy site/blog. It pretty much covers internet marketing strategy related stuff.

Come and check it out if you get time :-)
 
If you operate an affiliate program, this quick and easy solution can help your affiliates sell more of your productssoftware downloadssoftware downloads
 
Hi all, to answer your question: Yes, there is info talking about guaranteed happycampermlmleads.com lead mlm signups and it is worth reading. Great videos explaining everything about guaranteed happycampermlmleads.com lead mlm signups and most important, the money you will make. See you soon. Robert
 
Last month I made $12,124 with Google Adsense...Click
here for free on how you can do the same!

 
Easy to use software that allows even a complete beginner to create their own ebooktime clock softwaretime clock software
 
Hi there, I think you have a very good Blog, and I will have to Book Mark your blog. I have a web site that talks about mlm business opportunity cdr. If you have time maybe you can check it out at mlm business opportunity cdr
 
Hello! What at great blog you have going. I blog about network marketing internet business. I am also offering the eBook "Adsense Empire" to my visitors.... Stop by & get your copy today
 
http://corporatestrategicalliances.com/http://corporatestrategicalliances.com/Effective marketing tools, create a wave of traffic hungry for your product/services with this free set of marketing tools in the download area.

Grab your share of internet income with the most explosive discovery now free to download
 
I don't really agree with the contents of your post but I'm not about to start a flame war.

Take care.
movie file sharing
 
Great Blog. I enjoyed the information. Many marketers are solving the email blocking and filtering by discovering voice broadcasting. 100 percent of the messages get through! See call center at call center
 
Need sales lead's? I have found a proven
method anyone can implement to very easily earn an extra $500, $1,000 or even
$5,000+ every single week, with only 15 to 30 minutes of your time, and create a
significant ongoing monthly income. This method is a no-nonsense, set it and
forget it system, which will virtually run on 100% autopilot. People just like
you are earning enough money to quit their jobs within the first month. Stop
worrying about the bills, cancel your daily commute, never leave your family for
a job that is making someone else rich, and start living your life the way YOU
choose! New fail-safe system virtually runs 100% on autopilot. Just set it and
forget it!- Check it out sales lead 
sales lead
 
Dear friends, here is a really cool site that gives you a lot of FREE information that you can really use. Get FREE Daily and Weekly Foreign Exchange Outlook, Rates, Trends and Currency Shifts. Click here: FOREX
 
Hey Guys if you have been dumped by adsense or given the run around by Yahoo then you need to look at this, Fair Ads Network are only taken 10,000 members then they are closing the doors, I suggest that you take a look Now before its too late : http://fairadsnetwork.com/034704959097
 
Good blog - very interesting!! Thought you might like some excellent advice on driving extra traffic to your website/blog - I'm into borders book store and to drive traffic to my website I was searching around the internet for absolutely ages looking for SEO tools to fire my site up the Google search engine, then I signed up free for this superb traffic gereating tool - I now have loads of hits to my websites/blogs and my site is shooting up the Google rankings quicker than ever before.

Consequently this can see your affiliate commissions shoot through the roof - meaning lots of extra money coming in for you and your family. Sign up now - it's free.
. Sorry if this information is of no use to you but seeing as you have the anonymous feature enabled I thought I should share this free bit of essential promotional advice with you and would hope you would do the same for me.

I hope the tool will serve you equally as well as it has me!!

Best of luck, cheers for now, dave.
 
Great site about
work from home mlm business opportunity. I am bookmarking it.
Please also visit my site aboutwork from home mlm business opportunity
 
.





Hi matey

I'm sure it was this blog I was reading a while back where someone was looking for an SEO tool for driving more traffic and get more hits for their website.

Anyway, I was speaking to a techi guy at work who gets to know all the latest stuff and he uses the free Link Referral Program.

His website on affordable promotion site web amongst other stuff has seen traffic explode since he started using the Link Referral Program - consequently his affiliate sales commissions and business sales went through the roof PLUS his website increased in google ranking which was an added bonus!!

If anyone else has any good ideas for driving more traffic to blogs/websites then please share with your online business blog buddies. Ta ;-)










.
 
Hey, you have a great blog here! I'm definitely going to bookmark you!

I have a internet marketing consulting site/blog. It pretty much covers ##KEYWORD## related stuff.
 
2000% INTERNET PROFITS
 
.





Hi matey

I'm sure it was this blog I was reading a while back where someone was looking for an SEO tool for driving more traffic and get more hits for their website.

Anyway, I was speaking to a techi guy at work who gets to know all the latest stuff and he uses the free Link Referral Program at http://marketingexperts.wordpress.com.

His website on business home internet marketing opportunity amongst other stuff has seen traffic explode since he started using the Link Referral Program - consequently his affiliate sales commissions and business sales went through the roof PLUS his website increased in google ranking which was an added bonus!!

If anyone else has any good ideas for driving more traffic to blogs/websites then please share with your online business blog buddies. Ta ;-)


home based business
home based business opportunity
internet home based business
home based web business
best home based business
home based business idea
home internet based business opportunity
start home based business
best home based business opportunity
online home based business
income opportunity home based business
legitimate home based business
home based business lead
mlm home based business
starting a home based business
top home based business
christian home based business
home based small business
home based internet marketing business
free home based business
work at home based business opportunity
computer home based business
home based mlm business opportunity
make money home based business
home based business work
home based travel business
easy home based business
serious home based business opportunity
small home based business opportunity
online home based business opportunity
home based business for woman
business opportunity home based work from
work from home based business
new home based business opportunity
start a home based business on the internet
based business business home opportunity
legitimate home based business opportunity
home based sales business
home based jewelry business
best rated home based business
home based franchise opportunity business
successful home based business
home based business opportunity idea
profitable home based business
internet home based business idea
home based business work at home
best home based business idea
home based business opportunity work at home
free home based internet business
home based business tax deduction
top home based business opportunity
home based computer business opportunity
unique home based business opportunity
home based affiliate business
home based business opportunity uk
income opportunity home based business idea
based business home incredible opportunity
based best business home small
starting home based internet business
christian home based business opportunity
home based business merchant account
best home based internet business
based business home network opportunity
home based business marketing
based business home idea latest
home based business review
home based business directory
based business career home opportunity
work at home based internet business
based business home job opportunity
home based business trade secret
best home based business
home based business magazine
home based computer business idea
home based business startup idea
home based business for sale
home based business cash opportunity
home based business web site
most successful home based business









.
 
Good article. Thank you.
http://www.diazepam.350.com
 
This is good site to spent time on. allergy
 
Post a Comment

<< Home

Powered by Blogger