 |
|
|
|
|
|||||||
 |
|||||||||||
|
|||||||||||
|
Printing problems? |
|||||||||||
|
|
|
 |
||
 |
||
|
Evaluating the Quality of Localization Deliverables, Part I In April, the SimulTrans Localization Series featured two speakers: Emmanuel Uren, Manager of International Software Quality Assurance at Siebel Systems, Inc.; and Addison Phillips, Manager of Operations at SimulTrans. Phillips offered a holistic view of quality assurance as it relates to localization, while Uren discussed specific technical aspects of the QA process. Addison Phillips The Problem Companies are being driven to produce much more simultaneous shipments than in the past. Everyone is bent on delivering more languages with a faster turnaround time. One unfortunate result: The quality cycle gets squeezed. The Solution How do you address issues related to the need for faster turnaround and more languages? First, define what quality means in terms of your deliverables, then build a testing and development process to deliver quality product. As part of the solution, you must also build vendor relationships. This will allow you to more effectively "drive" vendor behavior and get what you want when you need it. Understanding and building vendor relationships is key to a successful localization process. Prepare Early Localization is 90% preparation and 10% perspiration. Projects that stray from this formula tend to go off-track. The need for time-saving steps such as internationalization soon becomes apparent. It becomes critical that you:
Get the Right People Involved Building global release mechanisms and planning for global release means getting the people who are both making and selling the product involved in your process. They should include representatives from:
If people such as international marketing representatives don't understand what kind of product will be released or what kind of product time cycles will occur, you'll have a hard time building a program to deliver those products. The Importance of Internationalization You may need to modify your developers' behavior. Be sure your developers are trained in localization and internationalization. This will allow you to talk about issues as they arise. If you don't foster internationalization training and localization awareness in your development community, products cannot be properly localized. Just today I was talking to a customer regarding Japanese localization of his product. He confessed that the localization process may not go well since, once again, his company barcoded essential items deep down inside the application. There's no way to get at it. This marks the fourth project I've worked on with this company and also the fourth time we've had this sort of conversation. Only now are they seriously exploring new methods of development. In addition to thorough training of your developers, you need to:
Before you reach the end of the development cycle, put some Japanese in the product and see if it blows up. Give it to your localization vendor and have them perform a pseudo translation. If you see problems with internationalization before you get to the end of the development cycle, you will have programmer dollars left to spend on repairs. On the other hand, if you've done the QA work on the product, built the product, and put the product in a box for U.S. companies, then you're left without any remaining budget or time. Apply all of this to your alpha codeor before. Horror Stories A client recently told me that things were going well and that his company had left 60 percent more room for text expansion during his product's development cycle. However, the word "scan" crept into the product. The word for "scan" in German, this company's target language, far exceeds the initial allotment. This proves that there are no hard-and-fast rules when it comes to text expansion. In addition, there's a plethora of web-like GUIs featuring visually pleasing text bars and graphical icons with words underneath them. Very pretty, very attractive user interfacesbut not localization friendly. These GUIs feature hard-coded strings that get dumped in under the graphic. While the text is centered in the English version, it now drops off to the right in the target language and may even run into the next icon. Or, if going into Japanese, the text may run four points taller and off into the bottom of the screen. By looking at these issues early on, you can avoid having to re-code sections of applications. Be aware of people who are new to the localization industry. If you're trying to foster connections with people who are considering localizing a product, talk to their developers about internationalization. Find out what your vendor's requirements are going to be. A number of people rely upon their past development experience; they don't understand what the issues are likely to be. As a result, something critical can derail the project. Machine translation can create another set of problems. Engineers often see translation as simply replacing the word for "dog" in English with the word for "dog" in German. Language doesn't work like that. Instead, you'll get string tables which are chopped up in little blocks and that need to be assembled at run time. You get problems with subject-verb agreement, terms of gender and the like. People tend to not understand that words move around. Complete Your Localization Kit Once you have product in hand, assembling the localization kit becomes the next critical step. The complete kit should contain:
A label on the CD sleeve may need to go in the box along with other items. Think about building your localization program around an entire product, not just pieces of a product. The localization kit should help you produce a completely localized product, one you can put in a box and ship to the customer in the targeted county. Internal style guides are also very important. If you have a guide that lays out the documentation in English, that should also be forwarded to the translators and desktop publishing team. If you have specific GUI design guidelines, the target language should adhere to those same guidelines. And you should build up internal memory relative to what goes in a localization kit. Instructions also need to be included. People need to know how to build the product and make the deliverable you expect. Consider the materials you're going to give to content developers, especially for software localization people who write documentation for you; they need to know everything there is to know about your product. Also, supply them with an image of the product. Think about what it is you want to get back. For example, you may want a zip file that contains an image CD-ROM that you're going to ship to your customer. You want that customer to simply unzip the file and have an "out-of-box" experience. If that's what you want your customer to have, then that's what you should ask your vendor to produce for you. Other issues to address in the instruction package: What do you want done? What do you expect to receive? When do you want it? Who do I, the vendor, call when something is wrong or I have a question? What do I do with these materials? How do I build this? How do I take it apart? And so on. Foster Vendor Relationships Communication between you and your vendor is critical. Without it, you will experience problems at some point. People who don't ask questions aren't taking a good look at the materials at hand. Visit your vendor regularly. Hold a kick-off meeting as well as a closing one. If possible, assign someone to your vendor's site. This presence can strengthen your relationship and help you understand what needs to happen to make the project a success. Hold regularly scheduled phone conferences and face-to-face meetings. And remember, project managers are the center of a localization experience. They are the people who make it happen. Once you understand your vendor, you can control what he or she is doing. For starters, insist on quality metrics from the beginning. Such metrics allow you to measure the results of what you do, outline what the scores are, and document how it works. Everyone is then in agreement regarding the validation methods, how long people have to do re-work, what the re-work cycle is going to look like, etc. A number of people don't think about these issues until after the project's start. They think they may find bugs and then ask you to fix them. Nobody sat down in advance and talked about how these sorts of problems are going to be addressed. It is critical because, when you get down to the end of the cycle, you're really going to be up against your deadline, up against the wall in terms of time and money. Your vendor should be part of your business cycle. You need him or her to understand that the due date is the due date, and that everybody is going to lose if you can't get your product to market. Your contracts, ideas, and expectations should be built around this kind of sharing. Build Your Testing Program Although building a testing program is the final step when preparing for localization, you need to start early. As part of teaching your QA people, inform them of international issues. If they're not testing your English release for international issues, they're not testing your globalization program. If you're uncertain whether or not your coders are producing international code, you will, unfortunately, find out when your vendor calls and says, ‘"Gee, I put German in here and I got back squares for all the characters." Your coders should include the international testing in their test cases, then provide those test cases to your vendors as part of your localization kit. Your vendors can then take that information and expand on it to test items in the target language. You also need to secure necessary resources, such as:
If you're working with client/server systems, the character sets of driver versions are critical to those systems. If you turn Oracle on with the Latin character set running, any Japanese will be unintelligible. It may be running Japanese and you may see all Japanese menus, but that won't help if you're storing Western European characters. You need the right piece of software throughout. The bigger the system, the more components and platforms you're trying to support, the bigger the problem and headache in trying to build a good test environment. Lastly, decide on methodologies. Choose what you're going to test in full and what things you're going to sample for quality. Sampling for quality means that you trust your vendor is doing a good job. It allows you to look at 10% of the work and see how well he or she is doing. If that 10% fails, then you look at another 10% and so on. If you test everything every time, and doing so times 12 languages, may prove difficult. Share with your vendor the scoring system you're going to use. Make sure you both agree on it. Then negotiate the requirements, i.e. major/minor bugs, what constitutes each, and what the fix cycles are. Scoring Systems If you don't have an internal standard, one available today is provided by LISA, the Localization Industry Standards Association. It defines everything in great detail and your vendors will most likely welcome its comprehensive nature. The LISA Standard allows you to:
It also offers easily agreeable definitions of acceptability. There are three levels of severity in various scoring systems. A certain number of error points can be picked up based on the severity of bugs found within a product. The three levels are:
All of this adds up to a point system. You end up with a score for the product in different areas of the software. This allows you to communicate effectively with your vendor after the project's close. "Your language was great, the product was fully functional, but the formatting was really, really messy." This allows you to point out the problem areas to your vendor. You can then focus on sensible groupings of errors. Bug Tracking and Regression Get your vendor involved in your bug tracking system. Share how you regress bugs, how you dispose of them, who does it, when it happens, and how you flag localization bugs. Then gain access to your vendor's bug tracking system. Find out what bugs they've got, how they're regressing the bugs, etc. Actual Testing In terms of your own testing, having language resources available is important. If you can't read the screen, you may have a hard time telling whether you're looking at Japanese or Chinese product. It could be even more difficult when looking at Dutch versus Norwegian versus Danish. Understand the software and documentation linkages and plan accordingly. You need to see how all of the pieces fit together. Plan for early samples, then have session plans. Plan for when the sessions are going to happen and tell your vendors about the sessions early on. Make sure everyone understands when the cycle happens and what goes into that cycle. Emmanuel Uren My undergraduate degree is in physics, so I spent a lot of time in a laboratory measuring currents, voltages, gamma count, and gamma rays. My first major occupation was as an operations researcher where I tried to be very precise about measuring dollars and what was, for example, the cost of sending a tanker somewhere. As a result of my background, half of this talk will be about measurement. As for quality, I find the concept to be very fuzzy. I'm European, of a certain age. I drive a 20-year-old Mercedes with 350,000 miles on it. I like to wear jeans. There's this sense of longevity, simplicity, efficiency that I'm attracted to. But when we get into software, we're certainly not talking about longevity. So it's best to offer a couple of definitions as they relate to software at the very start:
When it comes to software, a working definition for "quality" is, basically, the absence of bugs. This applies to all deliverables, Help, other documentation, and software (text strings and other localizables). Consequences of Low Quality Few can afford to release software that has no bugs. If a microeconomist or an operations researcher were t look at just when you should let the product out, they would be looking at the cost of development. It's more or less linear with time. The longer you spend on developing software (and tracking down bugs), the more money you're spending. In the meantime, you're losing sales. Microeconomics and cost Ideally, from an economical point of view, you would release the product at the lowest combined cost of the total unit cost lus the cost of returns and technical support. Internal Failures What are other costs made up of? Low quality is one of them. Within your own company, a low-quality product may cause you to incur costs related to:
External Failures Outside forces are also at work. Once a low-quality product is shipped, additional costs can include:
Methods of Measurement Metrics are often used in the QA world, although I've found very few useful measurements. One that is used routinely is KLOC, which refers to the number of bugs found per "thousand lines of code." People weigh this method over all kinds of programming languages, COBOL and C++. The first time you write something, you've got about 10 hours per 1,000 lines of code. If you start being systematic about the way you conduct QA, for example, just having a simply bug tracking system in your organization, can help you reduce bugs by a factor of 10. If you become more sophisticated, you can get down to one bug per 2,000 lines of code. That's the rate they use for some mission-critical sites, like spacecraft, etc. Things can get a little creepy because, say with Star Wars, they were talking about 30 million lines of codewhich means 15,000 errors. I don't find the KLOC measurement system to be very useful except to the development organization. Another method of measurement used is MTBF, or "mean time between failures." This lets users know how often a bug is going to show up or occur. Yet another measurement that has more utility than the previous two is "bug discovery rate." This utility refers to how many bugs you discover per day, per week, per month. Somewhat aligned to this is "cumulative fix rate," which refers to how many bugs your engineering department manages to fix per week, per day, or per hour. Conducting QA is like trying to find a needle in a haystackexcept you're not sure if there IS a needle there. Suppose you're working on some system and you found 500 bugs. So what? Are there 501 or 1,500 or 2000? Seeding Experiment This method of measurement has you deliberately introduce errors. If you put 100 errors into your software, your QA department may find instead 1,000 bugs. Perhaps 10 of those are the planted ones. In theory, you perhaps saved your team 10% of the work by introducing those 100 bugs. There are a number of statistical assumptions at work here and I don't know of anyone who's prepared to do this type of seeding. Coverage Analysis This measurement system has some utility but is impractical with very large systems. If used, it can provide you with a method of objective measurement. This can be helpful if someone, such as a project manager, asks you how well a product has been tested. Coverage analysis usually takes place in three steps. In Step 1, you "instrument the code. When you do this at the branch level, it becomes the case statement. For every branch of a case statement, you can give it a specific ID and do not disturb the logic of the program. In Step 2, you compile. Because you've put all these extra statements in it, the code is a bit bigger than the original and will probably run a little more slowly. To remedy this, you can write a trace file that says, against this set of tests, I executed these different branches. So you run a series of tests and you accumulate the results of all these various runs. Then you write a report that says, "These are the branches I hit and these are the ones I missed. With this test suite, I executed 20% of the program, or 40%." You are also in a position then to look at your code to see what you missed. You can add to your test suite and go back and run more tests, build up this hit and decrease that. Then, when someone asks you how thoroughly you tested, you can explain that you ran these test suites and covered 60% of the code. Now you need to achieve some type of objective measurement of how good or how thorough your testing is. It's very, very hard to get over 85%. The programmers themselves may be doing some testing and they usually get 30%. It can get incredibly difficult to start building from 50% up. Internationalization and Localization Issues Following are topics you should consider when testing non-U.S. versions:
Strategies for Maximizing Localization Quality Some suggestions to help prevent the creation of errors:
Detection of Errors The earlier you find a bug, the cheaper it is to fix. Test as early as possible. It helps to have substantial planning of test processes in place. You can do a substantial amount of static QA before you have anything to execute. It also helps to do the most difficult localization first, as in double-byte languages. Localize and automate whenever possible. What's very, very difficult in automation is handling the unexpected. You may need a fairly complicated piece of code that builds up the scenario in which you run tests. In building up that scenario, you may get a crash. And if you want to be able to come in Monday morning and say, "I ran these tests," you're going to have to be able to synchronize the output with what went on. So, if you crash in the middle of building up the test, you don't want all the other input going along and unsynchronizing everything. So automation can be very expensive and, in may cases, far more difficult that writing the original code. You can also plagiarize from domestic test plans and test cases. For linguistic testing, make the context of the string available to editors and proofer. Use back-translation on a sample basis. Back-translation On a sampling basis, e.g. 1-in-100, -500, -1000), have a native speaker translate (back) from the target language to the source language and compare the twice-translated result with the original. This should be done in addition to the traditional editing or proofing step in commercial translation. If they match, then the original translation is almost certainly okay; if they don't then there might be a problem. Another method of checking for errors is to mix parts from all locales as they become available. You can also compare localization functionalities in an effort to locate errors. Some companies conduct a "blitz" where everyone just bangs away at the product until all the errors are found and corrected. And, of course, there's beta testing. However, localized products seem to come along after the U.S. version, so this process is skipped. Remaining Mysteries Things I've run into and still don't know how to handle include:
|
||
