The following was written for the professional - software developer.

Pre-Replication Product Testing and Tweaking: Part 2

© 1999, by Terry E. Mercer, Pacific Buyers' Group
Published by Replication News, 1999

When creating a product, most programmers and developers seem to ignore two very critical things: 1) that other people use different equipment (hardware & software) that may alter how their particular program operates, and 2) that the majority of the computer users of most programs are novices (possibly knowing three or four programs to an intermediate level). Novice users flood technical support lines, bad mouth a wide variety of programs for a lack of user-friendliness, and often blame publishers for crashing a perfectly good computer when THAT PUBLISHER'S software was installed. The average user is able to effectively do an amazingly wide variety of inappropriate things to a seemingly flawless program: not reading messages, clicking mouse buttons (while the program is already trying to go somewhere), hitting the reset button because a program is taking too long to respond, and deleting a program through Windows Explorer because they didn’t like it, or don’t want it. These are among the most common reasons most software companies are busy with technical support calls. Many of these new end-users ends up throwing out the offending product in frustration. The average user reads directions as a LAST RESORT, as "modern" programs are assumed to be intuitive and interactive, which means instructions need to be written in such a fashion that the novice can clearly understand the information presented. Basic testing by a disinterested third party can save ANY company a lot of money, time, and reputation. The steps I describe will help maximize your customer's marketing efforts, ability to sell their product, their need for more CD's and yes, give you a greater opportunity for more money through both new orders and future testing.

The topic the software covers does NOT matter. All software products have four primary areas that must be considered and analyzed:

1) installation,
2) intended use,
3) common misuse,
4) uninstallation.

Before I explain what to look for in software, things to try and how you can best help your customer, it is important to setup at least a basic testing area. This can be a room or cubicle, to a whole building, depending on how thorough you want to be, and what your budget is. For the purpose of this article, I will describe an extremely effective, low-cost testing center.

Staffing: To effectively test a product you should have at minimum of two people. One must be very knowledgeable and comfortable with both hardware and software, as they will be responsible for keeping the hard drives going, backing up data and maintaining the component lists. The second person must be a computer novice. For example, a high school or college student, retired parent, anyone with very little knowledge. The beginner's job is to help your high-end person observe how "normal" people will try to operate the program. This "team" is likely to open up a whole new set of problems that were never previously considered by either the advanced user or the program developers. The key is not to think that the beginner is "stupid" or "not following instructions," it is learning what they do and don't do. These observations will provide valuable suggestions and recommendations for your client that will make their program better and help them sell more of the CD-ROM discs you replicate for them.

Equipment: When I set up a test center, I try to have at least three or four computer systems that start at the minimum requirements the developer or publisher plans to advertise on the packaging. You don't need the biggest or best, as the average user won't be using that. You will need good, solid, and upgradeable boxes that offer you the greatest degree of flexibility and future expansion in your test facility. Today, most software states that it requires a 486DX system or better. I often take these words to the extreme, starting with the lowest "common" system, a 486DX33 or DX66. I also try to have a 486DX133 and a Pentium 66, two very common CPUs with distinctive architecture. For added testing and programs that state that they require a "Pentium or better," I usually try to set up a Pentium 90, Pentium 166, Pentium 233, Pentium II 300, and a Pentium II 400. The reason I use BOTH a 300 and a 400 is that there are ECC (Error Correction Control) options in the 400 that don't exist in the other types of CPU's. Beginning with the Pentium II-350 (all 400's and better) ECC, RAM, true 100MHz front-side BUS speeds, and the AGP port all add up to a change in the rules and the program's action and reaction. I have seen programs that had some very strange and unpredictable results on high-end computers, which functioned perfectly on the lower-end systems. If you counted, that sounds like six to nine different computers, and it can be, in more thorough testing labs or centers there maybe 20 or more, but it is not required for the basics… or for low over head test centers. The basics can be dealt with effectively with only two or three systems with removable drives. Keep in mind that nowhere have I mentioned the AMD, Cyrix, Win Chip, Intel's Celeron, or IDT CPU's - some of which change the rules about how a program will operate, but seldom effect the four basics – which is the fastest, easiest, and most common cause of technical support. (At one time, nearly 80% of Microsoft’s technical support was with the basics, 15% on intermediate (actual) use, and less than 3% with OEM’s and reseller training, and only 2% on advanced use). Solving the basics is vital – it’s money in the pocket for all parties involved.

My test center uses 17" monitors because the larger monitors are easier on the eyes. I set up a switch box with one monitor, keyboard and mouse, capable of effectively switching between one to eight computer "boxes" at the same time. In addition, I use removable hard drive chassis for the boot file on my test systems. This setup offers me the greatest flexibility with the least cost and stress. I usually have a backup of each computer's operating system, the drivers required for that computer and a few different "common" programs on either a ZIP, Jazz, tape, or CD-ROM. This means that when a program "blows out a drive" - it can be quickly restored to its original settings with minimal down time. One set of hardware, such as a Pentium 166 system, a 3.5" floppy drive, 16X CD-ROM drive, and 32MB of RAM might have between two to five hard drives set up for it. One drive would have DOS 5.0 and Windows v3.1x, another would contain the Windows 95 upgrade (from Windows v3.1x), a different hard drive would contain the full version of Windows 95, the fourth would have Windows 98, and possibly one more with Windows NT. It is important to note that the upgrade versions are different than the full versions, which in turn are different than the OEM versions. All need to be tested if the program uses any strange or unique installation or removal programs. The hard drives don't have to be large. In most cases 540MB is sufficient. Anything over 2.1 Gigabytes is overkill for testing 99.9% of the software being sold today. It can be important to be able to test the FAT 16, FAT 32 and NTFS, but this usually isn't necessary for the primary four testing phases (unless the program is a utility that alters the FAT (File Attribution Tables) or Partition information). It is important to note that the test systems should use different brands of video cards, different types of modems, and varying amounts of RAM. Unless your customer requires testing on specific devices, floppy drives, CD-ROM drives, keyboards, monitors, speakers, joysticks, and mice have no effect on software 95% of the time and are an unnecessary expenditure of time and money.

Installation: Does it auto-run? If it does, what happens when the disc is put in AFTER installation? Any strange or tricky installation screens? What about installation or registration codes? Are they complicated or difficult to find? Will a fictitious code work? How will the installation program function on a system? Where will the files be placed on the hard drive? Is there a danger that the files (mostly DLL's) will be overwritten with the program on installation, thus causing another program to have problems? A self-contained program is always the safest and usually the best. How will the installation program function under different operating systems? Will the instructions on the packaging be adequately and accurately marked? A self-contained program is always the safest to use and generally the best.

Intended Use: Who will likely be purchasing and trying to use the program? Is it likely to conflict with other types of programs that advanced users will notice, but beginners will overlook? Does it effect file associations? Keep in mind, many fax, modem, web-related, and printer programs are loaded through the START-UP or the registry. With more RAM and multi-tasking capability, remember that novices often don't think about closing other programs before trying to install or run a new program. Will that have any ill effects on the new program or the user's computer? Once the program is installed, how does the end-user access it? Through the START/PROGRAMS menu? Is there an icon on the desktop? Is it auto-loaded? Does the program use "standard" key strokes and menu selections? Does it effect the shut down or restarting of the computer system? Is there an effective recovery system for power failures? Does the flow and process make sense? When the program is ended, does it release the memory it used? Does it clean up any temp files it created?

Unintentional Misuse: How will an end user likely try to use the program? This is nearly impossible to guess at. You almost have to watch a novice operate the new program. There are only three possible answers to this question: 1) everything is fine, 2) better documentation and on-screen instructions are necessary, or 3) the program needs to be changed in the areas that caused confusion. Be forewarned, at this point in the game, most programmers don't want to hear about any changes. However, clearly defined problems presented with several viable solutions can ease that stress. Generally with tact and practice, your staff can learn to break the news and accurately evaluate the level or degree of importance. Use a one to ten scale of importance. Learn to rate the likelihood of errors or problems.

Uninstalling: Does the program have its own un-install routine? Does "Add/Remove Program" see its routine? What happens if someone just deletes it from Explorer? Are any files deleted from the Windows or Windows/System folders deleted that effect-unrelated programs? Are the references to the program deleted or removed from the registry correctly? If associations are created on specific file types, are they: 1) left on the system with program errors when double clicking on a given file, 2) removed, so that the user needs to recreate associations to a different program, or 3) redirected to a different program, preferably the one it replaced? Finally, what happens to the end-user's DATA files during the uninstall? Are they deleted with or without warning? Is the data left on the drive? Can the data be accessed by another program?

The questions for these four basic problems seem limitless, but there are common questions and often simple answers. As the saying goes: Prior Planning Prevents Poor Performance. Testing and gathering the answers to the questions will make a pretty full day (or two) for a couple of product testers with reasonable equipment and resources at hand. What is their time worth? What is it worth to your customer to limit or eliminate problems prior to replication, printing, marketing, distribution and telephones ringing with unhappy customers?