The Refiner's Fire: Quality Assurance Testing for DVD
January 2000 |
DVD production is a complex process in which many different media types, like strands of wool, are woven into an intricate tapestry. Consequently, the slightest glitch can cause a domino effect that ends up as a major headache.
According to Randy Berg, business development manager for Rainmaker Digital Pictures of Burbank, California, "it can take eight hours to change one video clip. So one little problem equals one day."
And if one small glitch kills a day, you can imagine how many days a big problem could waste. As a result, Quality Assurance (QA) testing, though among the duller aspects of DVD development, is paramount to any title's successful completion. Every producer/developer and replicator has a Quality Assurance (QA) horror story he or she will tell you, though the names will be changed (or left out) to protect the innocent. Take for example the story about the DVD movie that had to be recalled after 100,000 discs were stamped, labeled, packaged, warehoused, and distributed. Or the non-QA'd DVD presentation that got into the hands of the CEO on the podium before anyone realized that it simply wouldn't play. "That one bit us big time," says the developer, who admitted under promise of anonymity to still having bad dreams about the experience.
When a developer lets an error-infected project go to pressing, the ramifications can become legal if the client is hard-nosed and unforgiving. QA problems rarely end up in court, however, because according to Berg, a good developer will almost always bite the bullet and make things right, even if that means re-doing the entire disc. "If a client never comes back, that's far more disastrous in the long run than anything else," says Berg. "It may cost us $5,000 to make repairs, but that client could be spending hundreds of thousands a year with us. Fixing a glitch gets us good will and the next job."
How can the legal liabilities, additional costs, and embarrassment of disc disasters be avoided? By taking Quality Assurance very seriously. Good QA requires a good attitude. Every developer should institute formal QA procedures, including guidelines and checklists. Instituting serious QA measures in a company can add 25-35 percent more time (and therefore money) to a project, Randy Berg estimates, but the effort will pay off in the long run, he insists. And the good news is, several software products are available to help DVD producers with this all-important job.
THE P & L OF QA
QA isn't fun, but somebody's got to do it. In some companies, that "somebody" is the QA manager, who may have a team of assistants. At Rainmaker, Berg says, "the whole organization does QC," starting even before the project begins. Quality Assurance, he argues, should start with the pre-planning stage and continue throughout the project cycle. The worst thing you can do is leave all the QA for the end of the project.
Chris Armbrust, owner of San Rafael, California-based Marin Digital, agrees. "QA should start up front, with the design, and be done in pieces, and as a concurrent track of the development process," he says. "If you wait till the end to QA, you're lost."
Chris Armbrust believes that it is helpful to think of Quality Assurance as a process having four or more basic levels or steps. Rainmaker uses a similar layered approach to QA, but puts more emphasis on QA's role in the planning stages. Armbrust highlights the following levels:
- Level 1: Check the physical parameters of the disc itself.
- Level 2: Check input elements and assets.
- Level 3: Check the encoded data streams: MPEG and AC-3.
- Level 4: Check the authoring, including navigation, interactivity, and usability.
At the lowest level, you have to address the physical properties of the disc itself, search out physical defects, and get right down to the pits and lands. This bit-by-bit checking is called disc validation. Thankfully, most of this chore is handled for DVD developers by the replicator.
At Level Two, the producer carefully checks and inventories all the project raw materials that have come in--videotapes, audio tapes, text files, graphics files, menus, photos, subtitles, art, and the like. Most of these materials can be spot-checked, but audio and video deserve closer attention.
Wise developers will head off problems down the road (in the encoding stage), says Berg, by thoroughly checking the video and audio assets when they first get them, noting any deficiencies and recommending special encoding treatment such as filtering, noise reduction, and preprocessing.
Level Three testing begins as soon as all the audio and video is encoded. MPEG-2 and AC-3 streams are checked for encoding problems such as mosquitoing, macroblocking, digital hits, skip frames, picture breakup, invalid DCT coefficients, and 3:2 pull-down errors. Among the software tools that can aid the developer with the stream-checking tasks are MpegRepair from Cupertino, California-based PixelTools, and MProbe from San Jose-based Interra Digital Video Technologies.
Wise developers will head off problems down the road by thoroughly checking the video and audio assets when they first get them, noting any deficiencies and recommending special encoding treatment such as filtering, noise reduciton, and preprocessing.
Interra's MProbe is a sophisticated MPEG bitstream analysis and compliance checker. MProbe users can employ random access mode to navigate through the bitstream and embedded parts, or they can automate the debugging process by queuing up streams to run unattended in batch mode. The tool's stream viewer provides a hierarchical data structure view, which helps the user maintain his or her point of reference while navigating through large and complex system bitstreams. MProbe comes in different versions and with different prices, starting at $1,995 for a six-month site license and going up to $14,995 for the no-holds-barred version.
Interra has also just introduced another tool that may also be useful with this phase of QA. Called SyncCheck, it is a lip sync measurement and analysis tool for MPEG bitstreams. Loss of lip sync is one of the most noticeable and annoying problems that DVD developers encounter, and anything that can help will no doubt be appreciated. SyncCheck allows the user to deconstruct an MPEG-2 Transport or Program down to its elementary video and audio components, and examine the timing relationships between them. The software tool's main window displays the sequence of video frames (thumbnails) and colored waveforms representing the various audio channels against a common time line. Pricing for the just-released product was not yet available at press time.
While MpegRepair from PixelTools is not exactly a tester per se, it includes a rudimentary analyzer. It is actually four tools in one: a variable bit rate MPEG-1 and MPEG-2 encoder, a decoder, a video analyzer, and a simple editing tool. "MpegRepair is like a pair of pliers," says PixelTools' market planning director Dick Kors. "It's not fancy but it's a great general purpose tool for identifying and repairing a broad spectrum of common problems."
According to Kors, "Ninety percent of MPEG-related DVD production problems occur because what the guy gives you isn't what he said it is. For example, what he says is a pure elemental stream may actually include transport," says Kors. MpegRepair allows you to analyze and confirm your raw material. "It's good for the developer who wants to quickly get everything going," says Kors. "The first thing you do is run your stuff through MpegRepair's built-in decoder. It's not real time, but it is robust and will play just about anything," he says. "So you can at least get an idea of what you've got." MpegRepair sells for $2,995.
At Level Four, the serious QA'er tests his authoring code to make sure it does what he meant it to. At this stage, things are checked such as audio track synchronization and identification, subpicture synchronization, and multiplexing. Also, more subjective things like navigation and interactivity are checked.
The worst and most obvious problem in DVD quality control is that different DVD players work differently.
One software tool that can help with this authoring testing phase is Interra's Surveyor. Billed as "a complete content analysis and quality control tool," Surveyor is also useful in Level Three testing, since along with its authoring content-related features, it includes all the stream checking capabilities of its MProbe sister product.
Surveyor can be used to check elementary video (MPEG) and audio (MPEG and Dolby Digital) bitstreams, navigational data and commands, copy protection settings, system and general parameters, and DVD file structure. It can analyze disc images, zero Directory structures, titles (VTS) and title sets, multiplexed VOBs, and Program Chains (PGCs). It is designed to be used throughout the entire production cycle, thus "allowing quality assurance tests to parallel the typical phases of DVD production," according to an Interra white paper. "Interra researchers found that the problem of locating errors in the latter stages of production was one of the more common and expensive difficulties DVD authors encounter," the white paper continues. "Thus, the software was planned to help authors identify--and then correct--potential troubles early on in a systematic way."
Surveyor will even test subjective things like navigation, by checking for valid program chains and logical links, though it is not foolproof in this task. "It can't tell whether or not you intended button A to jump to place B, but it will tell you if Button A goes nowhere or gets stuck in an endless loop," says Interra's marketing director Paul Collins.
Surveyor user Jeff Stabenau, president of New York City-based Crush Digital, likes the tool because it "provides a nice level of checking but can't be totally relied on," he says.
"QA software is like Artificial Intelligence," Stabenau continues. "The software is smart but there are some things that only you know about the project, and it can't read your mind. For example, just because it flags a violation, that doesn't necessarily mean that what it flags is actually a mistake or a problem. You then need an additional step; the problem needs interpretation. You need the human touch."
Surveyor costs $5,995, which may seem a hefty price to pay for a piece of software. Collins says its value more than justifies the price. "It will pay for itself the first time it saves your company from burning an unnecessary check disc," he says.
Surveyor is probably most useful to owners of Sonic Solutions DVD Producer authoring systems, since it is practically built in. The two companies have struck an integration agreement, and together have created a new DVD analysis file format that translates DVD specification protocols into DVD Producer's scripting language. This tight integration makes it easy for an author to see exactly where a trouble spot lies, says Collins. "Surveyor will go through the navigational structure the way DVD Producer created it. Then when it does a report, it gives you a bitmap thumbnail of the screen shot of the first frame of video, so you can instantly visually see where the problem is." When used with other authoring systems, Surveyor merely gives the user information, such as the title set number or a VOB number where the problem occurs.
Collins really likes the way Surveyor works within Sonic's DVD Producer, and he predicts that integration of QA testing utilities into authoring systems will be a future trend.
DVD's QA QUIRKS
While DVD projects share some of the quality control problems of CD-ROM, DVD also has some unique problems of its own. The worst and most obvious problem is that different DVD players work quite differently. "In DVD there's a standard spec, but there's not really a standard platform," says Marin Digital's Chris Armbrust. He says he recently completed a project that used 76 title sets, and even though the DVD spec allows a total of 99 title sets, he discovered that a certain player bombs at 38 title sets. Therefore, he had to go back and restructure the project so it used fewer title sets.
Armbrust recommends avoiding these potential pitfalls by minimizing the number of buttons and menus used in a project. "The more you try to do in the way of interactivity, the more you bump into the limits of some of the players," says Armbrust.
The upshot to all this is that a DVD developer can never be absolutely sure that his or her title will play on all the players in existence without actually trying the disc in every player. This is a daunting task to most DVD producers. "At the very least, you should take your title down to your local Circuit City store and try it in however-many players they have in stock," Armbrust recommends.
A more thorough but costly alternative to the Circuit City field test is taking your title to a software testing lab. One of the best known in the DVD realm is Intellikey Labs of Long Beach, California, whose clients include Disney, Universal, DreamWorks, Paramount, and Sony. The independent company has been offering QA testing services for DVD since 1996. "We usually find issues with the discs we get, so our clients are very appreciative," says company president Darrell Evers. Intellikey does player-compatibility testing using a bank of over 20 different DVD players, and it also does functionality testing using real human beings who actually sit down and view the discs. "It's often necessary for an outside facility to do functionality testing because the staff at the development house just gets too close to their material to stay objective," says Evers.
Intellikey offers developers a broad range of DVD console and DVD-ROM QA testing service "packages," at many different price points. You can choose Basic testing, Enhanced testing, Custom testing, or the Quick Test. There is an extra charge for Non-Interrupted Viewing testing.
FUTURE OF QA SOFTWARE
Clearly, Quality Assurance is a job so important that any software that addresses the problem is bound to be welcomed by DVD developers. There is obviously a need for more of this kind of software, and the need isn't likely to disappear any time soon. "DVD compliance issues will never go away because the spec is enormously complex," says Interra's Paul Collins.
Unfortunately, even the best QA software can't do everything. It will never be a miracle cure. "QA software like Interra's Surveyor is a promising idea, and Surveyor is good as far as it goes," says Jeff Stabenau. "It fills some of the cracks in the seams. But QA software is no substitute for a pair of human eyes."
At its most basic level, Quality Assurance includes checking the disc media to see that it has no physical defects, a task called disc validation. As Parker and Starrett astutely point out in CD-ROM Professional's CD-Recordable Handbook (Pemberton Press, 1996): "An unreadable mass-produced CD-ROM can be attributed to only two possible causes-bad disc or bad data." Traditionally, checking for bad discs (i.e., validating them) has become the responsibility of the replicator, while checking for bad data remains the sole responsibility of the producer/developer.
In the CD-ROM world, a whole mini-industry has sprung up to provide replicators with disc validation and testing hardware and software. A number of companies sell disc analyzers, which are serious, specialized pieces of electronic gear that start at $20,000 and go up to about $150,000. Although they are essentially CD-R-equipped computers, "these things don't look like computers; they look like lab equipment," says Carolyn Dobbin, senior vice president of CD Associates. They are usually rack-mounted in a replicator's lab-like facility, and it's not unusual for a replicator to have a whole rack of them. Analyzers check discs for arcane things like BLER, jitter, skew, bi-refringence, and cross-talk. Doug Carson Associates, CD Associates, Clover Systems, Audio Development USA, and Koch DigitalDisc all sell CD-ROM disc validation products, but some of them have been slow to release DVD versions of their products.
"The DVD spec came from a loose group of Japanese companies and it didn't specify testing parameters as closely as did the old CD spec from Philips and Sony. It was not thorough and it was late. That tied our hands behind us," says Dobbin.
CD Associates has recently released its DVD Plus analyzer, a product whose claim to fame is its "real-time video" capability. Dobbin explains that this product allows you to analyze the video while you are watching it in real time. The system simultaneously displays the video, the PI Error Rate, and the variable bit rate. "Formerly, if you saw a problem in the video, you didn't know whether it was a disc problem or an MPEG encoding problem," says Dobbin. "But with this system, the engineer can instantly tell if it's a disc problem."
Companies Mentioned in This Article
CD Associates, Inc.
15-A Marconi, Irvine, CA 92618; 949/588-3800; Fax 949/588-3805; http://www.cdassociates.com
Crush Digital Video
147 West 25th Street, 4th Floor, New York, NY 10001; 212/989-6500; Fax 212/645-9093; http://www.crushdv.com
Doug Carson Associates, Inc.
1515 East Pine, Cushing, OK 74023; 918/225-0346; Fax 918/225-1113; email: firstname.lastname@example.org; http://www.dcainc.com
324 East Bixby Road, Long Beach, CA 90807; 562/424-9129; Fax
Interra Digital Video Technologies
2001 Gateway Place, Suite 440W, San Jose, CA 95110; 408/573-1400; Fax 408/573-1430; email: email@example.com; http://www.interrainc.com
4340 Redwood Highway Suite 224,
San Rafael, CA 94903; 800/881-7982; Fax 415/507-9968; firstname.lastname@example.org; http://www.Marin-Digital.com
10721 Wunderlich Drive, Cupertino, CA 95014; 408/374-5327; Fax 408/374-8074; email: email@example.com; http://www.pixeltools.com
Rainmaker Digital Pictures
175 East Olive Avenue, #405, Burbank, CA 91502; 818/526-1500; Fax 818/953-5051; http://www.rainmaker.com
Mark Fritz (firstname.lastname@example.org), an EMedia contributing editor, is a consultant and freelance writer based in Bloomsburg, Pennsylvania.
Comments? Email us at email@example.com.