Chapter 11

Practical Methodology

Successfully applying deep learning techniques requires more than just a good

knowledge of what algorithms exist and the principles that explain how they

work. A good machine learning practitioner also needs to know how to choose an

algorithm for a particular application and how to monitor and respond to feedback

obtained from experiments in order to improve a machine learning system. During

day-to-day development of machine learning systems, practitioners need to decide

whether to gather more data, increase or decrease model capacity, add or remove

regularizing features, improve the optimization of a model, improve approximate

inference in a model, or debug the software implementation of the model. All these

operations are at the very least time consuming to try out, so it is important to

be able to determine the right course of action rather than blindly guessing.

Most of this book is about diﬀerent machine learning models, training algo-

rithms, and objective functions. This may give the impression that the most

important ingredient to being a machine learning expert is knowing a wide variety

of machine learning techniques and being good at diﬀerent kinds of math. In prac-

tice, one can usually do much better with a correct application of a commonplace

algorithm than by sloppily applying an obscure algorithm. Correct application of

an algorithm depends on mastering some fairly simple methodology. Many of the

recommendations in this chapter are adapted from Ng (2015).

We recommend the following practical design process:

•

Determine your goals—what error metric to use, and your target value for

this error metric. These goals and error metrics should be driven by the

problem that the application is intended to solve.

•

Establish a working end-to-end pipeline as soon as possible, including the

416

CHAPTER 11. PRACTICAL METHODOLOGY

estimation of the appropriate performance metrics.

•

Instrument the system well to determine bottlenecks in performance. Diag-

nose which components are performing worse than expected and whether

poor performance is due to overﬁtting, underﬁtting, or a defect in the data

or software.

•

Repeatedly make incremental changes such as gathering new data, adjusting

hyperparameters, or changing algorithms, based on speciﬁc ﬁndings from

your instrumentation.

As a running example, we will use the Street View address number transcription

system (Goodfellow et al., 2014d). The purpose of this application is to add

buildings to Google Maps. Street View cars photograph the buildings and record

the GPS coordinates associated with each photograph. A convolutional network

recognizes the address number in each photograph, allowing the Google Maps

database to add that address in the correct location. The story of how this

commercial application was developed gives an example of how to follow the design

methodology we advocate.

We now describe each of the steps in this process.

11.1 Performance Metrics

Determining your goals, in terms of which error metric to use, is a necessary ﬁrst

step because your error metric will guide all your future actions. You should also

have an idea of what level of performance you desire.

Keep in mind that for most applications, it is impossible to achieve absolute

zero error. The Bayes error deﬁnes the minimum error rate that you can hope to

achieve, even if you have inﬁnite training data and can recover the true probability

distribution. This is because your input features may not contain complete

information about the output variable, or because the system might be intrinsically

stochastic. You will also be limited by having a ﬁnite amount of training data.

The amount of training data can be limited for a variety of reasons. When your

goal is to build the best possible real-world product or service, you can typically

collect more data but must determine the value of reducing error further and weigh

this against the cost of collecting more data. Data collection can require time,

money, or human suﬀering (for example, if your data collection process involves

performing invasive medical tests). When your goal is to answer a scientiﬁc question

about which algorithm performs better on a ﬁxed benchmark, the benchmark

417

CHAPTER 11. PRACTICAL METHODOLOGY

speciﬁcation usually determines the training set, and you are not allowed to collect

more data.

How can one determine a reasonable level of performance to expect? Typically,

in the academic setting, we have some estimate of the error rate that is attainable

based on previously published benchmark results. In the real-word setting, we

have some idea of the error rate that is necessary for an application to be safe,

cost-eﬀective, or appealing to consumers. Once you have determined your realistic

desired error rate, your design decisions will be guided by reaching this error rate.

Another important consideration besides the target value of the performance

metric is the choice of which metric to use. Several diﬀerent performance metrics

may be used to measure the eﬀectiveness of a complete application that includes

machine learning components. These performance metrics are usually diﬀerent

from the cost function used to train the model. As described in section 5.1.2, it is

common to measure the accuracy, or equivalently, the error rate, of a system.

However, many applications require more advanced metrics.

Sometimes it is much more costly to make one kind of a mistake than another.

For example, an e-mail spam detection system can make two kinds of mistakes:

incorrectly classifying a legitimate message as spam, and incorrectly allowing a

spam message to appear in the inbox. It is much worse to block a legitimate

message than to allow a questionable message to pass through. Rather than

measuring the error rate of a spam classiﬁer, we may wish to measure some form

of total cost, where the cost of blocking legitimate messages is higher than the cost

of allowing spam messages.

Sometimes we wish to train a binary classiﬁer that is intended to detect some

rare event. For example, we might design a medical test for a rare disease. Suppose

that only one in every million people has this disease. We can easily achieve

99.9999 percent accuracy on the detection task, by simply hard coding the classiﬁer

to always report that the disease is absent. Clearly, accuracy is a poor way to

characterize the performance of such a system. One way to solve this problem is

to instead measure

precision

and

recall

. Precision is the fraction of detections

reported by the model that were correct, while recall is the fraction of true events

that were detected. A detector that says no one has the disease would achieve

perfect precision, but zero recall. A detector that says everyone has the disease

would achieve perfect recall, but precision equal to the percentage of people who

have the disease (0.0001 percent in our example of a disease that only one people in

a million have). When using precision and recall, it is common to plot a

PR curve

with precision on the

-axis and recall on the

-axis. The classiﬁer generates a score

that is higher if the event to be detected occurred. For example, a feedforward

418

CHAPTER 11. PRACTICAL METHODOLOGY

network designed to detect a disease outputs

ˆy

(

= 1

| x

), estimating the

probability that a person whose medical results are described by features

has

the disease. We choose to report a detection whenever this score exceeds some

threshold. By varying the threshold, we can trade precision for recall. In many

cases, we wish to summarize the performance of the classiﬁer with a single number

rather than a curve. To do so, we can convert precision

and recall

into an

F-score given by

F =

2pr

p + r

. (11.1)

Another option is to report the total area lying beneath the PR curve.

In some applications, it is possible for the machine learning system to refuse to

make a decision. This is useful when the machine learning algorithm can estimate

how conﬁdent it should be about a decision, especially if a wrong decision can

be harmful and if a human operator is able to occasionally take over. The Street

View transcription system provides an example of this situation. The task is to

transcribe the address number from a photograph to associate the location where

the photo was taken with the correct address in a map. Because the value of the

map degrades considerably if the map is inaccurate, it is important to add an

address only if the transcription is correct. If the machine learning system thinks

that it is less likely than a human being to obtain the correct transcription, then the

best course of action is to allow a human to transcribe the photo instead. Of course,

the machine learning system is only useful if it is able to dramatically reduce the

amount of photos that the human operators must process. A natural performance

metric to use in this situation is

coverage

. Coverage is the fraction of examples

for which the machine learning system is able to produce a response. It is possible

to trade coverage for accuracy. One can always obtain 100 percent accuracy by

refusing to process any example, but this reduces the coverage to 0 percent. For the

Street View task, the goal for the project was to reach human-level transcription

accuracy while maintaining 95 percent coverage. Human-level performance on this

task is 98 percent accuracy.

Many other metrics are possible. We can, for example, measure click-through

rates, collect user satisfaction surveys, and so on. Many specialized