Testing Blog
Watching movies to find localization bugs
Wednesday, March 12, 2008
By Sharon Zhou, Kirkland Client Test Lead
In December, Google Pack shipped 10 new languages in 10 new countries/regions including China Pack. This was in addition to the 30 languages Pack was all ready available in. Localization testing for these 10 languages is not trivial. The testing needs to be done very quickly by experts in the language who may not have seen the application before. Localization testing (LQA) can also be costly since it requires multiple external vendors, and the LQA schedule is highly sensitive to changes in the product schedule.
The process that has been followed so far has been to have each product documented by an engineer detailing the workflow to navigate to each area of the UI, the appropriate inputs to be entered at each step, and what should be expected. The documentation time is considerable, and changes with product changes. The vendors must each consume this documentation and become functional at using the product in order to navigate through the product. There are also challenges to get vendors the appropriate permissions to access our unreleased products, and to download them at the site where they work.
To minimize the test cost, the Pack test team has implemented a significant amount of automation across the entire product driving the UI. One feature of the automation harness is the ability to record movies. For the new 10 languages, the Pack team tried a new process of using the automation to drive the UI, recording movies of the product UI, and sending these movies to our vendors along with a top level test plan. To evaluate the new approach, we also asked them to fill out a survey to have a quantitative concept of how much time we can save, and hence how much cost we can reduce.
The survey results come back very positive and encouraging. We received valuable feedback on what vendors need to conduct a fast and efficient test pass. Overall, this experiment saved the vendors an estimated 25% of their time overall. It was just as effective as the previous process, but was much simpler for them to complete. Our next steps will be to drive more of the UI. If the automation can touch every page, link and dialog, it can replace the traditional LQA testing method of installing and running build as people perform functional testing.
3 comments
Labels
TotT
113
GTAC
61
James Whittaker
42
Misko Hevery
32
Code Health
31
Anthony Vallone
27
Patrick Copeland
23
Jobs
18
Andrew Trenk
13
C++
11
Patrik Höglund
8
JavaScript
7
Allen Hutchison
6
George Pirocanac
6
Zhanyong Wan
6
Harry Robinson
5
Java
5
Julian Harty
5
Adam Bender
4
Alberto Savoia
4
Ben Yu
4
Erik Kuefler
4
Philip Zembrod
4
Shyam Seshadri
4
Chrome
3
Dillon Bly
3
John Thomas
3
Lesley Katzen
3
Marc Kaplan
3
Markus Clermont
3
Max Kanat-Alexander
3
Sonal Shah
3
APIs
2
Abhishek Arya
2
Alan Myrvold
2
Alek Icev
2
Android
2
April Fools
2
Chaitali Narla
2
Chris Lewis
2
Chrome OS
2
Diego Salas
2
Dori Reuveni
2
Jason Arbon
2
Jochen Wuttke
2
Kostya Serebryany
2
Marc Eaddy
2
Marko Ivanković
2
Mobile
2
Oliver Chang
2
Simon Stewart
2
Stefan Kennedy
2
Test Flakiness
2
Titus Winters
2
Tony Voellm
2
WebRTC
2
Yiming Sun
2
Yvette Nameth
2
Zuri Kemp
2
Aaron Jacobs
1
Adam Porter
1
Adam Raider
1
Adel Saoud
1
Alan Faulkner
1
Alex Eagle
1
Amy Fu
1
Anantha Keesara
1
Antoine Picard
1
App Engine
1
Arham Jain
1
Ari Shamash
1
Arif Sukoco
1
Bartosz Papis
1
Benjamin Pick
1
Bob Nystrom
1
Bruce Leban
1
Carlos Arguelles
1
Carlos Israel Ortiz García
1
Cathal Weakliam
1
Christopher Semturs
1
Clay Murphy
1
Dagang Wei
1
Dan Maksimovich
1
Dan Shi
1
Dan Willemsen
1
Dave Chen
1
Dave Gladfelter
1
David Bendory
1
David Mandelberg
1
Derek Snyder
1
Diego Cavalcanti
1
Dmitry Vyukov
1
Eduardo Bravo Ortiz
1
Ekaterina Kamenskaya
1
Elliott Karpilovsky
1
Elliotte Rusty Harold
1
Espresso
1
Felipe Sodré
1
Francois Aube
1
Gene Volovich
1
Google+
1
Goran Petrovic
1
Goranka Bjedov
1
Hank Duan
1
