In July 1999, the SimulTrans Localization Seminar Series featured several speakers on the subject of software QA.

While localization often involves challenges that are common across all languages and international markets, localizing into Asian markets can be particularly challenging for a number of reasons. The most obvious challenge is dealing with the language itself. Multi-byte character sets and different levels of formality tend to make these projects a bit more complex. They often require more time, slower translation time, and more thorough review and quality assurance.

The speakers at the July seminar presented issues related to both the client and vendor sides of localized software QA.

Summary

Adam Jones discussed how quality is built in to the localization process at SimulTrans. Some of the issues that Adam described were the different types of testing that can take place, the tools used for each type, tips for successful testing scenarios, questions to ask before testing begins, and the testing that takes place at each SimulTrans site.

Doug Jacobs described the software testing process at Cobalt Networks, with a focus on Japanese. As part of his discussion, Doug described the Cobalt products, listed the generic testing phases, and discussed the problems encountered when testing Japanese products.

Gentry Watson listed the top ten mistakes he made while testing a large localized medical charting system. These problems ranged from misunderstandings about schedules, to lack of internationalization, and ill-equipped testers.

Localization Quality Assurance at SimulTrans

Adam Jones

Adam is Vice President of Customer Programs at SimulTrans, and has been directing their customer outreach programs for over five years. Adam regularly gives training presentations at conferences of the Society for Technical Communications, the American Translators Association, the Software Publishing Association, and other groups. He has provided customized training and consulting for a number of SimulTrans' clients in Writing for a Global Audience, Creating Localization Programs, Designing Websites, Implementing a Translation Tools Program, Managing In-Country Reviewers, and a variety of other topics.

Quality Checks During Localization

Throughout the localization process at SimulTrans, action is regularly taken to prevent problems. This means that very few problems actually get through to be found during testing. The following quality assurance steps are in place, and these steps are tracked with checklists as the project proceeds.

Interlinguistic Sampling

This process takes periodic samples of translated text and reviews it to assess the language quality.

Full Linguistic Review

This process evaluates all of the translated text to make sure that it was translated correctly.

Final Linguistic Review

This process checks the software and documentation at the end of the translation cycle to make sure that nothing was broken during the engineering or formatting phases.

Final Formatting Review

This process ensures that the documents don't have DTP problems, and can be printed or displayed properly (depending on the item).

Types of Testing

A number of types of tests are conducted for localized software. These are separate from the quality assurance steps that are performed as part of the localization process described above.

Internationalization Testing

This type of testing usually takes place before localization begins, to identify potential problems. Ideally, internationalization testing is done before the English version is finalized. Some of the issues that are checked during internationalization testing are character set problems (particularly for multi-byte characters); calendar, time, date, and currency formats; sorting problems; font problems; interactions with other libraries; embedded strings in code; and string expansion.

Internationalization testing takes place on the English product by running it on different operating systems, looking through the code, pseudo-translating text, and using some of the internationalization testing tools described below.

Tools

SimulTrans uses a suite of tools from OneRealm to evaluate code and identify internationalization problems. See the OneRealm web site for more information and software demos. (http://www.onerealm.com)

User Interface Testing

This is the most common type of testing that SimulTrans performs. Every software localization project goes through this testing process. When running user interface testing for each language version of an application, you look for items like truncated text, missing controls, problems with accelerator keys, problems with input and text fields, character set/font display problems, and input processing problems. (For example, when a user enters text in their language and using  their character sets, is it handled properly?)

Tools

A new application from Translation Craft (http://www.tcraft.com) is called Tool Proof. This tool works similarly to Catalyst in that it displays the user interface elements and dialog boxes, so that you can check to see if controls are sized correctly and that no other interface problems exist.

Functionality Testing

This form of testing mirrors the SQA process of the original language product. Basically, you go through all the functions to make sure they work. This means that you are not just looking at the user interface, but also saving files, printing, and running through any functions that the application performs. When testing functionality, special attention should be given to locale-specific components such as time and date formats, and text processing. Functionality tests must be run in a test environment that matches the target market environment. Hardware, client/server applications, operating systems, networking scenarios etc. should reflect actual operating conditions as closely as possible.

Functionality testing doesn't usually occur as a standard part of the localization process. Many people think that performing only user interface testing is enough, and that functionality should not be impacted by localization. When applications are internationalized well, the risk is certainly minimized. Unfortunately, about 90% of the software that SimulTrans deals with runs across functionality problems. If these problems aren't discovered during the testing phases, the end customer will be the one to find them.

Tools

Tools used for functionality testing tend to be proprietary to the company developing the product being localized. A company might have created some test grids and other tools for testing the English product. If the automated test scripts are well internationalized, they may be able to be used on localized software.

Compatibility Testing

Compatibility testing ensures that the software runs correctly with the target language operating system. If you've localized an application into French, you need to test it on a French version of the operating system. Applications frequently pull resources from the operating system, and operating systems may not be consistent in how they deal with things. Some applications are integrated into the operating system, browsers, or other system-wide components, and these applications need to be extensively tested with the matching language counterparts.

There may also be locale specific hardware issues, for example, in Japan there is a special version of the NEC9800 series of computers. These systems are very different from the ones used in the US. You need to test on the equipment that customers will actually use.

Tools

It's important to have all of the localized operating systems for whatever platforms and versions that you support. You will also need localized browsers, server applications, and any other applications that interact with yours during common operation.

Component Testing

Component testing refers to testing of help or HTML based items. Help is fairly simple to test; you check the links and jumps, and ensure that keywords and index entries haven't been removed or duplicated. Search routines, particularly in web sites, can be tricky because a word in English may have a whole set of synonyms in another language. Or there may be three words in English for a concept, but only one word for the same concept in another language.

Tools

HelpQA and HTML-QA by Translation Craft are great tools for analyzing how big the help system is, checking the links and jumps, identifying other problems, and warning you if there will be a compile problem due to localization. RoboHelp has tools that integrate well with localization. Proprietary tools are also developed to help protect functionality during localization.

Assuring Successful Testing

There are a number of issues you should focus on to achieve successful testing efforts. Discuss these issues with your localization vendor, and put them into practice internally as well.

QA as a Separate Function

The QA function should be staffed with people who are not working on the product's development. QA needs to be recognized as a unique and vital engineering effort, and one that requires separate resources. It's not enough to expect the engineer to compile, perform four hours of ad hoc testing, and call it quits.

Specialized Testing Labs

You should set up a testing lab with all the  equipment, hardware, and software that you need. Supply it with the various platforms and operating systems required. You might have a client/server application with an NT server, and you'll need to test target languages for it. It's important to have computers set up so that testers can view the English and localized versions of the product side by side.

Your lab should mimic the full environment, and this may mean providing specialized hardware.  If you are developing a printer driver, it's not enough to merely look at the UI. You need to send files to the printer to see what happens. If your localization vendor will be doing some of this testing, you'll need to provide all of the hardware required to access hardware interface messages. SimulTrans has received printers, phone switches, mice, video cards, audio boards, etc. that were required for testing.

Some companies develop emulators to fake the functionality of hardware and firmware during development. These emulators can be shared with localization vendors for testing. If you are going to be creating a lot of localized versions and you have a complicated environment, you might want to consider investing engineering resources into some sort of emulator so that you can send off localization testing work more easily. It's always a hard decision between doing the testing at your own company because you have the hardware and software, and sending the testing to a localization vendor, because they have the language operating systems, hardware, and software.

Segmented Networks

SimulTrans sets up full testing networks for different projects, which are separate from the standard network. This allows the testing to take place in it's own environment to more closely simulate a real work environment.

Target Language Operating Systems and Support

Language versions of operating systems are fairly simple to get. If you are a member of the Microsoft Developer Network, you can request the international version at no charge. In the CD binder you can find the French version of Windows NT, or the German version of Windows 98. You can even pull these versions out and test your application on them before localization to see if anything breaks. Membership in the Network costs about $3,000 per person using the license, but it's worth it. The Apple Developer Connection handles things similarly. Sun has great internationalized and localized versions of Solaris, and Oracle has a developer network for localized versions of Oracle Server.

Managing the various language versions can be tricky. One easy way to do it is to re-image machines before each testing cycle. SimulTrans maintains clean images for all the popular operating systems and languages that they deal with frequently. They can therefore simply install the image onto the computer rather than spending hours installing and configuring one particular setup (for example, German NT). Using this imaging method saves a lot of time.

Formalized Bug Tracking and Version Control

People sometimes get a little less serious about testing when it comes to localized versions than they were with the source language version, and this can cause problems. You need to track and report bugs in an organized way, because you'll frequently need to go back to the original developer to get resolution.

Version control can also be an issue, because various versions of files with fixes can go flying back and forth between your company and the localization vendor very quickly. Things can get confusing without a defined version control procedure in place.

Testers Must Speak Target Language

It's not of much use to have an English speaking person test a Chinese application, and try to interpret the error messages that appear. Testers should know the language so that they can detect truncations or other language-related problems in the application, and so that they can advise engineering about how to make the fixes.

Questions to Ask

Here are some questions to ask as you are setting up a testing process for localized software.

• Where should testing be performed?
• What types of testing do you need?
• Are testing resources and script tools available?
• How should bugs be reported?
• Whose bug tracking system will be used?
• Who is responsible for resolving issues? (For example, is the localization vendor responsible for fixing internationalization problems, or merely for reporting them?)

Testing at SimulTrans

Localization testing takes place in all of the SimulTrans offices. User interface testing tends to be done where there are native speakers of the target language and engineers who can work in unison. But various offices specialize in other types of testing. Sending testing to another site is useful because it keeps the testing effort completely separate from the engineering effort, and this helps ensure impartiality.

The Mountain View office specializes in internationalization testing, and so internationalization jobs from all over the world are sent there. The Dublin office has a functionality test center where a number of people run through test scripts and routines in a variety of environments to check functionality.

SimulTrans also sends testers out to client sites when the environment and equipment can't easily be provided or replicated.

Localization Quality Assurance at Cobalt

About the Presenter

Doug Jacobs

Doug Jacobs is the SQA Lead Engineer at Cobalt Networks. Since he joined Cobalt in 1998, he has been involved with testing all Cobalt products, and is working to better develop web-based automation tools to assist Cobalt's QA department. He has two and a half years experience in the QA field, including a year at 3Com where he worked in mobile products on domestic, internationalization, and localization projects. Doug holds degrees in Computer Science and Japanese Studies.

Cobalt Products

Doug began his presentation by describing the Cobalt products that he works on.

The Qube is intended as a small workgroup or departmental server. It supports web sites, email groups, discussion groups, and a few other features. The Qube.2 also has a modem and integrated firewall and network address translation options. The Qube is localized for Japan, and was Cobalt's first localized product. The Japanese version shipped 3 months after the English, and that occurred about a year ago.

The RaQ is designed for the ISP market. Like the Qube, the RaQ is an email and web server, however, it can also do virtual hosting. This allows you to have many IP numbers on a single rack, with  many sites per IP. You can mix and match depending on how heavy the site usage is. Customers have put up to 1,000 sites on a single rack. The RaQ has also been localized for Japan, and it's selling very well there.

The next two product lines are not localized.

Cobalt Cache is a server that catches web traffic, and it runs on either the Qube or the RaQ. The purpose of the product is to cache sites that have been accessed, so that when another user goes there, they look at the chached information rather than actually traveling out across the web to the site. This reduces WAN bandwidth so that you don't use your T1 or ISPN line as much. Cobalt Cache has been very popular in Europe where speed is more of a problem than it is here in the US.

Network Attached Storage (NAS) is a dedicated file sharing server, with hardware enhancements over the Qube, and SCSI support.

Test Phases

Doug then described the testing process at Cobalt. There are six phases of testing for the English base products:

Early or Quiet Testing

This phase may be called something else in other companies, and is not an official part of the testing cycle. Quiet testing is the informal process of checking out a feature as the engineering group completes it. For example, an email configuration tool might be ready, and the testing group takes the product and doesn't pay attention to any features other than that one. Some engineers may be uncomfortable with this type of testing, because the product is not in a polished state, and they don't want bug reports issued when it's not an official release.

Alpha Testing

Alpha testing is the start of the formal test cycle. At this stage, all functionality has been completed. It's not necessarily working, but no strings are missing or anything like that. The software has reached the stage where testing can be attempted. Product begins to ship to external sites at the alpha testing stage, and this plays an important part in the localization testing.

Beta Testing

At the beta stage, the product should be close to complete. All functionality is in, and all serious and critical problems have been resolved. The beta spin is basically minor changes to UI strings, a few last minute fixes, and perhaps an enhancement or a new button.

Gold Master Process

The Golden Master or GM Candidate is the release that the engineers think is ready to ship. The testing group takes it and when they can't find any problems that need to be fixed, the candidate becomes the GM Release.

The last item to be tested before actual production begins is to take the first unit (called the First Article) off the assembly line and run a quick round of tests on it to make sure that the right software is on it, in the right language, etc.

Testing Types

When evaluating localized products, Cobalt performs the following types of tests.

User Interface Testing

This testing is similar to what Adam described. It is a review of how the strings, buttons, and fields of the web-based tool look.

Functionality Testing

This testing ensures that all of the services that a particular unit is supposed to provide actually work. For example, on an email server, testers make sure that they can actually send and receive email. Cobalt tests to see how things work with various clients and mail systems. There are occasional conflicts with various combinations of clients and services. They also make sure that you can do things like successfully dragging a Macintosh file onto one of the servers, and pull it off using a PC.

Compatibility Testing

Since the products provide web services, it is critical that they are compatible with the major browsers for major platforms. Cobalt looks at Internet Explorer and Netscape under most Microsoft operating systems and on the Macintosh. There is some support for Linux since the servers are running Linux internally as their operation system. You can use language kits under Linux for Netscape. But when testing for localization, Cobalt sticks with Netscape Japanese on a system that's actually running in Japanese.

Compatibility testing runs through the text fields, input fields and buttons to make sure that everything works under various configurations.

Stress Testing

As the name implies, this type of testing evaluates how well the product stands up to heavy use. Cobalt has a number of boxes set up that run a script which simulates traffic to and from the servers. Some of these run for several months. Customers will inevitably surprise you with the ways they'll push your product to its limit, but you can at least try to anticipate heavy workload conditions.

Testing Process for Localized Software

When testing localized products, the process is slightly different than with English. There is no quiet testing stage, so everything begins at alpha.

Cobalt attempts to run testing in a localized environment as much as possible. They contract native speakers to come in for the testing, and they also work with SimulTrans. Beta feedback from market placements is very important, because you can only mimic the real life environment to a certain degree. The internet can't be replicated within a lab very easily. The best beta testers are noisy; that way if they find a problem, you'll definitely hear about it. Cobalt is lucky enough to have two resellers in Japan that provide good feedback.

Problems in Japanese Products

When testing the Japanese versions of their products, Cobalt came across some interesting problems.

Language Support Limitations

One of the biggest problems found was that language support for certain client applications such as FTP or Telnet did not exist. For example, Win 95J and Win 98J come with Telnet and FTP clients, however, they aren't localized. If you try to use the Microsoft FTP client to fetch files off a Cobalt server and those files have Japanese file names, the FTP client will destroy the file. It can't handle the double byte file names. Cobalt had to do some research about how the Japanese deal with this problem. Some people said that they don't create file names with Japanese characters. Other people use a different application to transfer files. So there was not one set solution.

A related challenge was that English Linux is used on the inside of the products, regardless of the language being shipped. Some commands (such as TAR which is used for archiving files) treat all files as single byte. This created problems when doing file backups, because double byte file names were missing in the archives.

Conditional Text

One of the features of the Qube.2 presented a message which contained conditional information such as a server name. This was implemented in a way that did not allow for repositioning of the conditional information within the sentence, which meant that the message read fine in English, but that it was essentially gibberish in Japanese.

Locale-Specific Features

As you develop a localized product, you may find feature requirements that are specific to individual locales. For example, one of the screens on Japanese Cobalt products has a button that the English version does not have. This was due to the fact that in Japan, there are a number of different formats for DNS configuration files. This is not an issue in the U.S. The Japanese product therefore contains a drop down list box  which lists various ISPs so that users can select the format they want.

Learning Curve

When testing the English product, the testers may know the thing inside and out. When bringing in people with language skills, however, there is a learning curve to get the testers up to speed. While they have the language familiarity, they probably don't have the experience or technical knowledge of the product itself.

Question and Answer Session

Question
What languages do your products currently support?

Answer
Currently, Cobalt only localizes into Japanese, but by the fall they should be moving into Korean, German, and French. Having started with a multi byte language should help make the transition to Korean a bit smoother since they've had some exposure to possible problems.

Question
Do you create a separate test plan for Japanese?

Answer
Cobalt uses almost the same test plan. There are a small number of extra tests that differentiates the plan from the English, but Cobalt is trying to combine the two. For instance, they might take an English server to a Japanese subnet and have people try to send and receive email in Japanese and create Japanese file names. The features should work correctly. Therefore, it would be simpler to run from one test plan.

Question
At what stage is the plan used? Do you pas the plan along to the beta site or SimulTrans, or do you do it mostly in-house during the quiet test stage?

Answer
Cobalt plays it by ear depending on what is being tested. For DNS, it was difficult because a hierarchy of DNS servers was required. Networks in the U.S. are generally Class C, but in Japan they deal with much smaller networks (such as 4-bit subnets.)  In this situation, they asked their Japanese resellers to thoroughly test DNS. Cobalt tries to guide their beta testers, but they also let them run pretty freely to see how the product acts in the real world.

Question
Which do you do first; internationalization testing or localization testing?

Answer
Cobalt does both simultaneously.  The QA group tries to do preventative internationalization tests like pushing Japanese files up onto an English Qube to see what happens.

One of Cobalt's goals is to produce an international product, so the next thing they might do is to roll the new DNS functionality out into all of the Qube products. Another goal is to change the UI to use more icons and less text. They are trying to find images which are readily understood around the world, and that is a challenge.

Question
What is the desired delta between the English and Japanese releases?

Answer
Ideally, they would like the products to come out on the same day. You may remember the Next computer; you put it on your desk and the first thing it asked was what language you wanted to use. The ultimate goal is to have one product and users merely select their language instead of buying a specific language product. Better yet would be to have language selection on the fly so that if you have one administrator who is Japanese and another who is Korean, they could use the same box and select the language for the UI when they needed to use it.

Currently, however, the English and Japanese products are completely separate.

Top Ten Mistakes in a Rush Japanese Project

About the Presenter

Gentry Watson

Gentry L. Watson is the Program Manager for the Internationalization Engineering Department at Sun Microsystems. His responsibilities at Sun include project management of current I18N activities, liaison with both internal and external I18N organizations, and development of business plans for future I18N activities. He has had the opportunity to work in many different functional areas of the computer industry. He has experience in IBM Mainframe system programming, UNIX system programming, industry associations and standards organization, product management and marketing, I18N/L10N development and project management. Gentry earned both his B.S.E.E. and M.S.E.E. at the University of Kansas.

Project Background

Gentry began his discussion by describing the environment in which a problematic globalization project took place. The project involved creating a Japanese version of a medical information system. It was developed by a healthcare software division of a large corporation. The system was like an electronic version of a chart at the end of a patient's hospital bed; it included information about clinical tests, lab tests, etc. Roughly three to five million lines of code were involved.

Unfortunately, nobody in the organization really understood how it all worked. Each person understood the piece they'd worked on, but time had passed since the system was developed, and knowledgeable people had left.

There were approximately 50 installations of the system across the world. It was popular in the U.S. market and sold through a distributor in Europe. No 8-bit or localization testing was performed at headquarters because localization was handled by the European distributor. The development group never saw localized versions. At this stage, Gentry was brought in to do a crash project: funding had just been received to develop a Japanese product.

When Gentry joined the organization, he'd spent years developing operating systems, and as a systems programmer. He'd also worked with UNIX International (an industry association) tying to develop internationalization requirements for UNIX. Gentry had no hands-on internationalization or localization experience when he arrived. He had an interest in language, but his prior job had been marketing product manager for operating systems at the same company.

Mistakes

Mistake Number 1—Never Let a Physician Write Code

Some of the system's code was developed by physician's who unfortunately thought they were super programmers. The situation was helped when the physicians were sent out to talk to customers instead of writing code. More software engineers were brought in to do the work that the physicians previously provided.

Mistake Number 2—Lack of Engineering's Endorsement

When Gentry walked in to the project, some of the engineers and physicians told his team that they could never do a Japanese project. The initial environment was pretty negative. The team was staffed primarily from people outside the organization, and this meant that they had a lot of difficulty in getting information about the product design. One of the physicians who remained in the engineering function was particularly stubborn about sharing information. Gentry eventually came to the point where he would go into the man's cubicle, sit down, and say "I'm not leaving until you answer my questions. Tell me when you want the questions." He would then sit and stare at him until the physician gave up and talked.

Luckily for the team, there was one experienced engineer in the group who decided that he wanted to be associated with the project. Without him, there would have been no hope for success.

Because of this difficulty in breaking through the walls that the existing engineers erected, a lot of on-the-job training had to take place. Everyone had to learn as they actually created the product. 

Mistake Number 3—Misunderstandings About Schedules

Have you ever worked on a project that if you'd known at the beginning what you knew at the end, you'd never have signed up for it? Gentry had to deliver a product in seven months, while the 8-bit version had taken three years to develop. The only reason the team succeeded was because of the incredible level of motivation and commitment by all the team members. Perhaps they were crazy.  The lesson learned from this was that better international planning was required, and Gentry took on this responsibility for later projects.

Mistake Number 4—Not Having the Right Skills

The Japanese system required extensive X-Windows work, in which the team did not have experience. They read a lot of manuals and learned while doing, but the real solution was hiring experienced programmers.

Mistake Number 5—Lack of a Glossary

The clinical information system contained about 45,000 terms. Many people on the project didn't understand the English terms, let alone having to deal with Japanese. The terms were spread throughout about 650 configuration files.

The initial process used for translation was inefficient and required intensive editing. Translated files were received back from a number of different translators, and the team edited discrepancies when the same term was translated differently in different files. Also, configuration text was translated when it should have been left alone, and this had to be corrected.

The solution was to create a text extraction and insertion tool. This tool was used to generate a glossary of terms, and later to propagate the translated terms back in to the files. The new process pulled terms out of the 650 files to generate one giant glossary. This glossary was then translated and pushed back into the files.

Mistake Number 6—Using a Poorly Qualified Translation Vendor

There was no effort made to find a qualified translation company or to find translators who specialized in medical terminology. The only reason the project succeeded despite this was that the Japanese salespeople provided a group of nursing students to help ensure the correctness of terminology.

Mistake Number 7—Not Having Qualified Testers

Given the nature of the product, quality assurance was critical. When the testing phase was reached, the team worked from the following base; there was a set of English test plans which were extremely difficult to follow, there were a group of English speaking nurses who were familiar with testing the product, and there were Japanese sales support personnel and helpful Japanese customers to provide linguistic feedback. Given this scenario, it was difficult to know how to best test the Japanese system.

One option considered was to have the "friendly" customers perform functionality tests, but Gentry was concerned that the problems wouldn't be documented well enough to allow for debugging, and that the speed of communication might not meet schedule demands.

The approach taken was to test in the U.S., placing English and Japanese systems side by side.  Both functionality and UI issues were tested. The English-speaking nurses ran the English test plans with Japanese-capable engineers on call. If anything was different between the screens, one of these engineers would write the problem report, diagnose the problem, and attempt to fix it as quickly as possible.

The nurses doing the functionality testing were all volunteers; no one was put on the project under duress. This turned out to be a very good thing, because the testing effort was very stressful since they didn't really know what they were looking at. The team taught them how to look up words in Japanese dictionaries, but they didn't know Japanese and had to struggle with it.

The Japanese sales people did touch testing to provide feedback about terminology, but they were learning the product at the same time. This was the extent of the linguistic QA. Luckily, the translation quality was good.

Mistake Number 8—Not Following the Company's Development Process

In order to meet the seven-month deadline, many process and quality shortcuts were taken as the project progressed. The first customer was a very specific installation, and some functionality had to be dropped in order to make delivery. The development and testing teams raised understandable concerns about delivering a risky product, but the team understood that it was essential to get the product out in order to remain in the game.  The international team tried to comply with process standards, but they still couldn't shed their renegade image.

Mistake Number 9—Localizing Before Internationalizing

Gentry knew when he went in to the project that localizing without internationalization was the wrong approach. The engineering organization wanted to keep out of the project completely because they didn't want to put the U.S. product at risk. As a result, the Japanese baseline was separate from the U.S. baseline, and this created maintenance problems almost immediately. This remained a problem for some time, until eventually an internationalized base product was released.

Mistake Number 10—No Long Term Commitment to the International Marketplace

After the initial Japanese product was released, the international market somehow became second priority. The team could never get funding to develop any other languages. The Japanese version had perhaps seven installations before the division was sold. The acquiring company was not interested in international markets, but did agree to support the Japanese customers and the European market as it previously existed.

Summary

The conclusion of all this was for Gentry to have gained a lot of hands-on internationalization and localization experience. The team made many mistakes and Gentry plans to use those learning experiences when dealing with problems in future projects.

Question and Answer Session

Question
Did you consider scripting?

Answer
The company was evaluating the use of scripting for both the English and Japanese versions, but a lot of the testing was graphical; you'd access windows and choose options from pull-down lists etc. Tools did not seem to be available at the time, but it would have been a great idea.

Question
Did you have free text entry, or was most of the entry from pull-down lists?

Answer
Most of the tests were scenarios doing pull-down lists and fill in forms from pre-selected options. This made it much easier for a non-native speaker to run the tests. We wrote some specific tests for entering Japanese characters, and these tests were done by the Japanese-capable developers and the salespeople in Japan.

Question
Did the test plan check the edge cases or stress the system in some way, or did they just check basic functionality?

Answer
The test cases just tested functionality, however, the touch testing and the engineering testing tried to add some unpredictable behaviors to see what would happen.

Question
How do you pick the right translation company?

Answer
You have to look at your product and language set, and then look for a vendor that specializes in your specific market and/or language set. You should go out and do the background work for your specific needs, and not trust any one company to give you the answers. There may also be some business issues associated with your decision. The best provider may not be the cheapest. There is no one right answer.

Question
What is the biggest testing mistake that SimulTrans has encountered?

Answer
People seem to underestimate the overall impact of cross-platform support for server applications. People want to ship products that magically run on multiple platforms, but they don't want to have to test it. This leads to poor customer satisfaction, and greater overall expense in trying to fix things later instead of testing and fixing before shipment.