From ffcc94df00c3c36dbcb48967c4cc243290765aef Mon Sep 17 00:00:00 2001 From: wassname <1103714+wassname@users.noreply.github.com> Date: Fri, 14 Aug 2026 21:13:11 +0800 Subject: [PATCH] evidence: full text for sculley 2015, lones 2021, domingos 2012 These three held hand-transcribed excerpts (381 / 914 / 573 words) with a header claiming extraction was too hard. It was not: jina reads the sculley and lones PDFs, and plain pdftotext (no -layout) reads all 3 columns of the domingos CACM PDF in order. Now 5736 / 15285 / 8180 words. --- .../domingos_2012_few_useful_things.md | 1279 ++++++++++++++++- docs/evidence/lones_2021_ml_pitfalls.md | 492 ++++++- .../sculley_2015_hidden_technical_debt.md | 161 ++- 3 files changed, 1883 insertions(+), 49 deletions(-) diff --git a/docs/evidence/domingos_2012_few_useful_things.md b/docs/evidence/domingos_2012_few_useful_things.md index 87a69c6..3e9d288 100644 --- a/docs/evidence/domingos_2012_few_useful_things.md +++ b/docs/evidence/domingos_2012_few_useful_things.md @@ -1,38 +1,1277 @@ Source: https://homes.cs.washington.edu/~pedrod/papers/cacm12.pdf (author's copy; CACM doi:10.1145/2347736.2347755) -Title: "A Few Useful Things to Know About Machine Learning" — Pedro Domingos, Communications of the ACM, Oct 2012, vol. 55 no. 10 -Fetched-via: PDF downloaded from Domingos' UW page, pages transcribed by hand from the rendered pages (3-column CACM layout defeats text extraction) -Fetch-status: verbatim excerpts +Title: "A Few Useful Things to Know About Machine Learning" -- Pedro Domingos, Communications of the ACM, Oct 2012, vol. 55 no. 10 +Fetched-via: curl the pdf, then pdftotext WITHOUT -layout, 2026-08-14. jina returns an empty body on this 16MB image-heavy pdf, and pdftotext -layout clips the 3-column text; plain pdftotext reads all 3 columns in order. +Fetch-status: verbatim, full article; page furniture (running heads, page numbers) left inline -# A Few Useful Things to Know About Machine Learning (excerpts) +review articles +doi:10.1145/ 2347736.2347755 -Standfirst and intro (p. 78) — the paper's stated purpose is writing down ML folk knowledge: +Tapping into the “folk knowledge” needed to +advance machine learning applications. +by Pedro Domingos -> Tapping into the "folk knowledge" needed to advance machine learning applications. +A Few Useful +Things to +Know About +Machine +Learning +Machine learning systems automatically learn -> Several fine textbooks are available to interested practitioners and researchers (for example, Mitchell and Witten et al.). However, much of the "folk knowledge" that is needed to successfully develop machine learning applications is not readily available in them. As a result, many machine learning projects take much longer than necessary or wind up producing less-than-ideal results. Yet much of this folk knowledge is fairly easy to communicate. This is the purpose of this article. +programs from data. This is often a very attractive +alternative to manually constructing them, and in the +last decade the use of machine learning has spread +rapidly throughout computer science and beyond. +Machine learning is used in Web search, spam filters, +recommender systems, ad placement, credit scoring, +fraud detection, stock trading, drug design, and many +other applications. A recent report from the McKinsey +Global Institute asserts that machine learning (a.k.a. +data mining or predictive analytics) will be the driver +of the next big wave of innovation.15 Several fine +textbooks are available to interested practitioners and +researchers (for example, Mitchell16 and Witten et +al.24). However, much of the “folk knowledge” that -Key-insights box (p. 78): +78 -> developing successful machine learning applications requires a substantial amount of "black art" that is difficult to find in textbooks. +comm unicatio ns o f the ac m -"It's Generalization that Counts" (p. 80): +| o c to ber 201 2 | vo l . 5 5 | no. 1 0 -> The fundamental goal of machine learning is to generalize beyond the examples in the training set. [...] Doing well on the training set is easy (just memorize the examples). The most common mistake among machine learning beginners is to test on the training data and have the illusion of success. +is needed to successfully develop +machine learning applications is not +readily available in them. As a result, +many machine learning projects take +much longer than necessary or wind +up producing less-than-ideal results. +Yet much of this folk knowledge is +fairly easy to communicate. This is +the purpose of this article. -> Contamination of your classifier by test data can occur in insidious ways, for example, if you use test data to tune parameters and do a lot of tuning. (Machine learning algorithms have lots of knobs, and success often comes from twiddling them a lot, so this is a real concern.) +key insights + M achine learning algorithms can figure -"Overfitting Has Many Faces" (p. 81): +out how to perform important tasks +by generalizing from examples. This is +often feasible and cost-effective where +manual programming is not. As more +data becomes available, more ambitious +problems can be tackled. -> What if the knowledge and data we have are not sufficient to completely determine the correct classifier? Then we run the risk of just hallucinating a classifier (or parts of it) that is not grounded in reality, and is simply encoding random quirks in the data. This problem is called *overfitting*, and is the bugbear of machine learning. When your learner outputs a classifier that is 100% accurate on the training data but only 50% accurate on test data, when in fact it could have output one that is 75% accurate on both, it has overfit. + M achine learning is widely used in -> Everyone in machine learning knows about overfitting, but it comes in many forms that are not immediately obvious. +computer science and other fields. +However, developing successful +machine learning applications requires a +substantial amount of “black art” that is +difficult to find in textbooks. -"Feature Engineering Is The Key" (p. 84): + T his article summarizes 12 key lessons -> At the end of the day, some machine learning projects succeed and some fail. What makes the difference? Easily the most important factor is the features used. +that machine learning researchers and +practitioners have learned. These include +pitfalls to avoid, important issues to focus +on, and answers to common questions. -> First-timers are often surprised by how little time in a machine learning project is spent actually doing machine learning. But it makes sense if you consider how time-consuming it is to gather data, integrate it, clean it and preprocess it, and how much trial and error can go into feature design. Also, machine learning is not a one-shot process of building a dataset and running a learner, but rather an iterative process of running the learner, analyzing the results, modifying the data and/or the learner, and repeating. +Image by agsa ndrew/Sh utt erstock.co m -"More Data Beats a Cleverer Algorithm" pull quote (p. 84): +Many different types of machine +learning exist, but for illustration +purposes I will focus on the most +mature and widely used one: classification. Nevertheless, the issues I +will discuss apply across all of machine learning. A classifier is a system that inputs (typically) a vector +of discrete and/or continuous feature values and outputs a single discrete value, the class. For example, +a spam filter classifies email messages into “spam” or “not spam,” +and its input may be a Boolean vector x = (x 1,…,x j,…,x d), where x j = 1 if +the j th word in the dictionary appears +in the email and x j = 0 otherwise. A +learner inputs a training set of examples (x i, y i), where x i = (x i,1 , . . . , +x i, d) is an observed input and y i is the +corresponding output, and outputs +a classifier. The test of the learner is +whether this classifier produces the +correct output yt for future examples +xt (for example, whether the spam +filter correctly classifies previously +unseen email messages as spam or +not spam). + +Learning = Representation + +Evaluation + Optimization +Suppose you have an application that +you think machine learning might be +good for. The first problem facing you +is the bewildering variety of learning algorithms available. Which one to use? +There are literally thousands available, +and hundreds more are published each +year. The key to not getting lost in this +huge space is to realize that it consists +of combinations of just three components. The components are: +˲˲ Representation. A classifier must +be represented in some formal language that the computer can handle. +Conversely, choosing a representation for a learner is tantamount to +choosing the set of classifiers that it +can possibly learn. This set is called +the hypothesis space of the learner. +If a classifier is not in the hypothesis +space, it cannot be learned. A related +question, that I address later, is how +to represent the input, in other words, +what features to use. +˲˲ Evaluation. An evaluation function (also called objective function + +or scoring function) is needed to distinguish good classifiers from bad +ones. The evaluation function used +internally by the algorithm may differ from the external one that we want +the classifier to optimize, for ease of +optimization and due to the issues I +will discuss. +˲˲ Optimization. Finally, we need +a method to search among the classifiers in the language for the highest-scoring one. The choice of optimization technique is key to the +efficiency of the learner, and also +helps determine the classifier produced if the evaluation function has +more than one optimum. It is common for new learners to start out using +off-the-shelf optimizers, which are later replaced by custom-designed ones. +The accompanying table shows +common examples of each of these +three components. For example, knearest neighbor classifies a test example by finding the k most similar +training examples and predicting the +majority class among them. Hyperplane-based methods form a linear + +o cto b e r 2 0 1 2 | vo l . 55 | n o. 1 0 | c om m u n icat ion s of t he ac m + +79 + +review articles +Table 1. The three components of learning algorithms. + +Representation + +Evaluation + +Optimization + +Instances + +Accuracy/Error rate + +Combinatorial optimization + +K-nearest neighbor + +Precision and recall + +Greedy search + +Support vector machines + +Squared error + +Beam search + +Hyperplanes +Naive Bayes +Logistic regression + +Likelihood + +Branch-and-bound + +Posterior probability +Information gain + +Continuous optimization +Unconstrained + +Decision trees + +K-L divergence + +Gradient descent + +Sets of rules + +Cost/Utility + +Conjugate gradient + +Propositional rules + +Margin + +Quasi-Newton methods + +Logic programs + +Constrained + +Neural networks + +Linear programming + +Graphical models + +Quadratic programming + +Bayesian networks +Conditional random fields + +Algorithm 1. Decision tree induction. +LearnDT (TrainSet) +if all examples in TrainSet have the same class y* then +return MakeLeaf(y*) +if no feature xj has InfoGain(xj ,y) > 0 then +y* ← Most frequent class in TrainSet +return MakeLeaf(y*) +x* ← argmaxxj InfoGain(xj, y) +TS0 ← Examples in TrainSet with x* = 0 +TS1 ← Examples in TrainSet with x* = 1 +return MakeNode(x*, LearnDT(TS0), LearnDT(TS1)) + +combination of the features per class +and predict the class with the highest-valued combination. Decision +trees test one feature at each internal +node, with one branch for each feature value, and have class predictions +at the leaves. Algorithm 1 (above) +shows a bare-bones decision tree +learner for Boolean domains, using +information gain and greedy search.20 +InfoGain(xj, y) is the mutual information between feature xj and the class y. +MakeNode(x,c0,c1) returns a node that +tests feature x and has c0 as the child +for x = 0 and c1 as the child for x = 1. +Of course, not all combinations of +one component from each column of +the table make equal sense. For example, discrete representations naturally +go with combinatorial optimization, +and continuous ones with continuous optimization. Nevertheless, many +learners have both discrete and continuous components, and in fact the +80 + +comm unicatio ns o f the acm + +day may not be far when every single +possible combination has appeared in +some learner! +Most textbooks are organized by +representation, and it is easy to overlook the fact that the other components are equally important. There is +no simple recipe for choosing each +component, but I will touch on some +of the key issues here. As we will see, +some choices in a machine learning +project may be even more important +than the choice of learner. +It’s Generalization that Counts +The fundamental goal of machine +learning is to generalize beyond the +examples in the training set. This is +because, no matter how much data +we have, it is very unlikely that we will +see those exact examples again at test +time. (Notice that, if there are 100,000 +words in the dictionary, the spam filter described above has 2100,000 pos- + +| o c to ber 201 2 | vo l . 5 5 | no. 1 0 + +sible different inputs.) Doing well on +the training set is easy (just memorize +the examples). The most common +mistake among machine learning beginners is to test on the training data +and have the illusion of success. If the +chosen classifier is then tested on new +data, it is often no better than random guessing. So, if you hire someone +to build a classifier, be sure to keep +some of the data to yourself and test +the classifier they give you on it. Conversely, if you have been hired to build +a classifier, set some of the data aside +from the beginning, and only use it to +test your chosen classifier at the very +end, followed by learning your final +classifier on the whole data. +Contamination of your classifier by +test data can occur in insidious ways, +for example, if you use test data to +tune parameters and do a lot of tuning. (Machine learning algorithms +have lots of knobs, and success often comes from twiddling them a lot, +so this is a real concern.) Of course, +holding out data reduces the amount +available for training. This can be mitigated by doing cross-validation: randomly dividing your training data into +(say) 10 subsets, holding out each one +while training on the rest, testing each +learned classifier on the examples it +did not see, and averaging the results +to see how well the particular parameter setting does. +In the early days of machine learning, the need to keep training and test +data separate was not widely appreciated. This was partly because, if the +learner has a very limited representation (for example, hyperplanes), the +difference between training and test +error may not be large. But with very +flexible classifiers (for example, decision trees), or even with linear classifiers with a lot of features, strict separation is mandatory. +Notice that generalization being +the goal has an interesting consequence for machine learning. Unlike +in most other optimization problems, +we do not have access to the function +we want to optimize! We have to use +training error as a surrogate for test +error, and this is fraught with danger. (How to deal with it is addressed +later.) On the positive side, since the +objective function is only a proxy for +the true goal, we may not need to fully + +review articles + +Data Alone Is Not Enough +Generalization being the goal has another major consequence: Data alone +is not enough, no matter how much +of it you have. Consider learning a +Boolean function of (say) 100 variables from a million examples. There +are 2100 − 106 examples whose classes +you do not know. How do you figure +out what those classes are? In the absence of further information, there is +just no way to do this that beats flipping a coin. This observation was first +made (in somewhat different form) by +the philosopher David Hume over 200 +years ago, but even today many mistakes in machine learning stem from +failing to appreciate it. Every learner +must embody some knowledge or assumptions beyond the data it is given +in order to generalize beyond it. This +notion was formalized by Wolpert in +his famous “no free lunch” theorems, +according to which no learner can +beat random guessing over all possible functions to be learned.25 +This seems like rather depressing +news. How then can we ever hope to +learn anything? Luckily, the functions +we want to learn in the real world are +not drawn uniformly from the set of all +mathematically possible functions! In +fact, very general assumptions—like +smoothness, similar examples having similar classes, limited dependences, or limited complexity—are +often enough to do very well, and this +is a large part of why machine learning has been so successful. Like deduction, induction (what learners do) +is a knowledge lever: it turns a small +amount of input knowledge into a +large amount of output knowledge. +Induction is a vastly more powerful +lever than deduction, requiring much +less input knowledge to produce useful results, but it still needs more than +zero input knowledge to work. And, as +with any lever, the more we put in, the +more we can get out. +A corollary of this is that one of the +key criteria for choosing a representation is which kinds of knowledge are +easily expressed in it. For example, if +we have a lot of knowledge about what +makes examples similar in our do- + +main, instance-based methods may +be a good choice. If we have knowledge about probabilistic dependencies, graphical models are a good fit. +And if we have knowledge about what +kinds of preconditions are required by +each class, “IF . . . THEN . . .” rules may +be the best option. The most useful +learners in this regard are those that +do not just have assumptions hardwired into them, but allow us to state +them explicitly, vary them widely, and +incorporate them automatically into +the learning (for example, using firstorder logic21 or grammars6). +In retrospect, the need for knowledge in learning should not be surprising. Machine learning is not +magic; it cannot get something from +nothing. What it does is get more +from less. Programming, like all engineering, is a lot of work: we have to +build everything from scratch. Learning is more like farming, which lets +nature do most of the work. Farmers +combine seeds with nutrients to grow +crops. Learners combine knowledge +with data to grow programs. +Overfitting Has Many Faces +What if the knowledge and data we +have are not sufficient to completely +determine the correct classifier? Then +we run the risk of just hallucinating +a classifier (or parts of it) that is not +grounded in reality, and is simply encoding random quirks in the data. +This problem is called overfitting, and +is the bugbear of machine learning. +When your learner outputs a classifier that is 100% accurate on the training data but only 50% accurate on test +data, when in fact it could have output + +one that is 75% accurate on both, it +has overfit. +Everyone in machine learning +knows about overfitting, but it comes +in many forms that are not immediately obvious. One way to understand +overfitting is by decomposing generalization error into bias and variance.9 +Bias is a learner’s tendency to consistently learn the same wrong thing. +Variance is the tendency to learn random things irrespective of the real signal. Figure 1 illustrates this by an analogy with throwing darts at a board. A +linear learner has high bias, because +when the frontier between two classes +is not a hyperplane the learner is unable to induce it. Decision trees do not +have this problem because they can +represent any Boolean function, but +on the other hand they can suffer from +high variance: decision trees learned +on different training sets generated by +the same phenomenon are often very +different, when in fact they should be +Figure 1. Bias and variance in +dart-throwing. + +Low +Variance + +High +Variance + +High +Bias + +Low +Bias + +Figure 2. Naïve Bayes can outperform a state-of-the-art rule learner (C4.5rules) even +when the true classifier is a set of rules. + +Bayes   + +80 +Test-Set Accuracy (%) + +optimize it; in fact, a local optimum +returned by simple greedy search may +be better than the global optimum. + +C4.5 + +75 +70 +65 +60 +55 +50 +10 + +100 + +1000 + +10000 + +Number of Examples + +o c to b e r 2 0 1 2 | vo l. 55 | n o. 1 0 | c om m u n ic at ion s of t he acm + +81 + +review articles +the same. Similar reasoning applies +to the choice of optimization method: beam search has lower bias than +greedy search, but higher variance, because it tries more hypotheses. Thus, +contrary to intuition, a more powerful +learner is not necessarily better than a +less powerful one. +Figure 2 illustrates this.a Even +though the true classifier is a set of +rules, with up to 1,000 examples naive Bayes is more accurate than a +rule learner. This happens despite +naive Bayes’s false assumption that +the frontier is linear! Situations like +this are common in machine learning: strong false assumptions can be +better than weak true ones, because +a learner with the latter needs more +data to avoid overfitting. +Cross-validation can help to combat overfitting, for example by using it +to choose the best size of decision tree +to learn. But it is no panacea, since if +we use it to make too many parameter +choices it can itself start to overfit.17 +Besides cross-validation, there +are many methods to combat overfitting. The most popular one is adding +a regularization term to the evaluation +function. This can, for example, penalize classifiers with more structure, +thereby favoring smaller ones with +less room to overfit. Another option +is to perform a statistical significance +test like chi-square before adding new +structure, to decide whether the distribution of the class really is different with and without this structure. +These techniques are particularly useful when data is very scarce. Nevertheless, you should be skeptical of claims +that a particular technique “solves” +the overfitting problem. It is easy to +avoid overfitting (variance) by falling +into the opposite error of underfitting +(bias). Simultaneously avoiding both +requires learning a perfect classifier, +and short of knowing it in advance +there is no single technique that will +always do best (no free lunch). +A common misconception about +overfitting is that it is caused by noise, +a Training examples consist of 64 Boolean features and a Boolean class computed from +them according to a set of “IF . . . THEN . . .” +rules. The curves are the average of 100 runs +with different randomly generated sets of +rules. Error bars are two standard deviations. +See Domingos and Pazzani10 for details. +82 + +communicatio ns o f th e acm + +like training examples labeled with +the wrong class. This can indeed aggravate overfitting, by making the +learner draw a capricious frontier to +keep those examples on what it thinks +is the right side. But severe overfitting +can occur even in the absence of noise. +For instance, suppose we learn a Boolean classifier that is just the disjunction of the examples labeled “true” +in the training set. (In other words, +the classifier is a Boolean formula in +disjunctive normal form, where each +term is the conjunction of the feature +values of one specific training example.) This classifier gets all the training +examples right and every positive test +example wrong, regardless of whether +the training data is noisy or not. +The problem of multiple testing13 is +closely related to overfitting. Standard +statistical tests assume that only one +hypothesis is being tested, but modern learners can easily test millions +before they are done. As a result what +looks significant may in fact not be. +For example, a mutual fund that beats +the market 10 years in a row looks very +impressive, until you realize that, if +there are 1,000 funds and each has a +50% chance of beating the market on +any given year, it is quite likely that +one will succeed all 10 times just by +luck. This problem can be combatted +by correcting the significance tests to +take the number of hypotheses into +account, but this can also lead to underfitting. A better approach is to control the fraction of falsely accepted +non-null hypotheses, known as the +false discovery rate.3 +Intuition Fails in High Dimensions +After overfitting, the biggest problem +in machine learning is the curse of +dimensionality. This expression was +coined by Bellman in 1961 to refer +to the fact that many algorithms that +work fine in low dimensions become +intractable when the input is highdimensional. But in machine learning it refers to much more. Generalizing correctly becomes exponentially +harder as the dimensionality (number +of features) of the examples grows, because a fixed-size training set covers a +dwindling fraction of the input space. +Even with a moderate dimension of +100 and a huge training set of a trillion +examples, the latter covers only a frac- + +| o cto ber 201 2 | vo l . 5 5 | no. 1 0 + +tion of about 10−18 of the input space. +This is what makes machine learning +both necessary and hard. +More seriously, the similaritybased reasoning that machine learning algorithms depend on (explicitly +or implicitly) breaks down in high dimensions. Consider a nearest neighbor classifier with Hamming distance +as the similarity measure, and suppose the class is just x1 ∧ x2. If there +are no other features, this is an easy +problem. But if there are 98 irrelevant +features x3,..., x100, the noise from +them completely swamps the signal in +x1 and x2, and nearest neighbor effectively makes random predictions. +Even more disturbing is that nearest neighbor still has a problem even +if all 100 features are relevant! This +is because in high dimensions all +examples look alike. Suppose, for +instance, that examples are laid out +on a regular grid, and consider a test +example xt. If the grid is d-dimensional, xt’s 2d nearest examples are +all at the same distance from it. So as +the dimensionality increases, more +and more examples become nearest +neighbors of xt, until the choice of +nearest neighbor (and therefore of +class) is effectively random. +This is only one instance of a more +general problem with high dimensions: our intuitions, which come +from a three-dimensional world, often do not apply in high-dimensional +ones. In high dimensions, most of the +mass of a multivariate Gaussian distribution is not near the mean, but in +an increasingly distant “shell” around +it; and most of the volume of a highdimensional orange is in the skin, not +the pulp. If a constant number of examples is distributed uniformly in a +high-dimensional hypercube, beyond +some dimensionality most examples +are closer to a face of the hypercube +than to their nearest neighbor. And if +we approximate a hypersphere by inscribing it in a hypercube, in high dimensions almost all the volume of the +hypercube is outside the hypersphere. +This is bad news for machine learning, +where shapes of one type are often approximated by shapes of another. +Building a classifier in two or three +dimensions is easy; we can find a reasonable frontier between examples +of different classes just by visual in- + +review articles +spection. (It has even been said that if +people could see in high dimensions +machine learning would not be necessary.) But in high dimensions it is difficult to understand what is happening. This in turn makes it difficult to +design a good classifier. Naively, one +might think that gathering more features never hurts, since at worst they +provide no new information about the +class. But in fact their benefits may +be outweighed by the curse of dimensionality. +Fortunately, there is an effect that +partly counteracts the curse, which +might be called the “blessing of nonuniformity.” In most applications +examples are not spread uniformly +throughout the instance space, but +are concentrated on or near a lowerdimensional manifold. For example, +k-nearest neighbor works quite well +for handwritten digit recognition +even though images of digits have +one dimension per pixel, because the +space of digit images is much smaller +than the space of all possible images. +Learners can implicitly take advantage of this lower effective dimension, +or algorithms for explicitly reducing +the dimensionality can be used (for +example, Tenenbaum22). +Theoretical Guarantees +Are Not What They Seem +Machine learning papers are full of +theoretical guarantees. The most common type is a bound on the number of +examples needed to ensure good generalization. What should you make of +these guarantees? First of all, it is remarkable that they are even possible. +Induction is traditionally contrasted +with deduction: in deduction you can +guarantee that the conclusions are +correct; in induction all bets are off. +Or such was the conventional wisdom +for many centuries. One of the major +developments of recent decades has +been the realization that in fact we can +have guarantees on the results of induction, particularly if we are willing +to settle for probabilistic guarantees. +The basic argument is remarkably +simple.5 Let’s say a classifier is bad +if its true error rate is greater than ε. +Then the probability that a bad classifier is consistent with n random, independent training examples is less +than (1 − ε)n. Let b be the number of + +One of the major +developments of +recent decades has +been the realization +that we can have +guarantees on the +results of induction, +particularly if we +are willing to settle +for probabilistic +guarantees. + +bad classifiers in the learner’s hypothesis space H. The probability that at +least one of them is consistent is less +than b(1 − ε)n, by the union bound. Assuming the learner always returns a +consistent classifier, the probability +that this classifier is bad is then less +than |H|(1 − ε)n, where we have used +the fact that b ≤ |H|. So if we want this +probability to be less than δ, it suffices +to make n > ln(δ/|H|)/ ln(1 − ε) ≥ 1/ε (ln +|H| + ln 1/δ). +Unfortunately, guarantees of this +type have to be taken with a large grain +of salt. This is because the bounds obtained in this way are usually extremely loose. The wonderful feature of the +bound above is that the required number of examples only grows logarithmically with |H| and 1/δ. Unfortunately, most interesting hypothesis spaces +are doubly exponential in the number +of features d, which still leaves us +needing a number of examples exponential in d. For example, consider +the space of Boolean functions of d +Boolean variables. If there are e possible different examples, there are +2e possible different functions, so +since there are 2d possible examples, +d +the total number of functions is 22 . +And even for hypothesis spaces that +are “merely” exponential, the bound +is still very loose, because the union +bound is very pessimistic. For example, if there are 100 Boolean features +and the hypothesis space is decision +trees with up to 10 levels, to guarantee +δ = ε = 1% in the bound above we need +half a million examples. But in practice a small fraction of this suffices for +accurate learning. +Further, we have to be careful +about what a bound like this means. +For instance, it does not say that, if +your learner returned a hypothesis +consistent with a particular training +set, then this hypothesis probably +generalizes well. What it says is that, +given a large enough training set, with +high probability your learner will either return a hypothesis that generalizes well or be unable to find a consistent hypothesis. The bound also says +nothing about how to select a good +hypothesis space. It only tells us that, +if the hypothesis space contains the +true classifier, then the probability +that the learner outputs a bad classifier decreases with training set size. + +o c to b e r 2 0 1 2 | vo l. 55 | n o. 1 0 | c om m u n ic at ion s of t he acm + +83 + +review articles +If we shrink the hypothesis space, the +bound improves, but the chances that +it contains the true classifier shrink +also. (There are bounds for the case +where the true classifier is not in the +hypothesis space, but similar considerations apply to them.) +Another common type of theoretical guarantee is asymptotic: given infinite data, the learner is guaranteed +to output the correct classifier. This +is reassuring, but it would be rash to +choose one learner over another because of its asymptotic guarantees. In +practice, we are seldom in the asymptotic regime (also known as “asymptopia”). And, because of the bias-variance trade-off I discussed earlier, if +learner A is better than learner B given +infinite data, B is often better than A +given finite data. +The main role of theoretical guarantees in machine learning is not as +a criterion for practical decisions, +but as a source of understanding and +driving force for algorithm design. In +this capacity, they are quite useful; indeed, the close interplay of theory and +practice is one of the main reasons +machine learning has made so much +progress over the years. But caveat +emptor: learning is a complex phenomenon, and just because a learner +has a theoretical justification and +works in practice does not mean the +former is the reason for the latter. + +A dumb algorithm +with lots and lots +of data beats +a clever one +with modest +amounts of it. + +Feature Engineering Is The Key +At the end of the day, some machine +learning projects succeed and some +fail. What makes the difference? Easily the most important factor is the +features used. Learning is easy if you +have many independent features that +each correlate well with the class. On +the other hand, if the class is a very +complex function of the features, you +may not be able to learn it. Often, the +raw data is not in a form that is amenable to learning, but you can construct features from it that are. This +is typically where most of the effort in +a machine learning project goes. It is +often also one of the most interesting +parts, where intuition, creativity and +“black art” are as important as the +technical stuff. +First-timers are often surprised by +how little time in a machine learning +project is spent actually doing ma84 + +communicatio ns o f th e ac m + +| o c to ber 201 2 | vo l . 5 5 | no. 1 0 + +chine learning. But it makes sense if +you consider how time-consuming it +is to gather data, integrate it, clean it +and preprocess it, and how much trial +and error can go into feature design. +Also, machine learning is not a oneshot process of building a dataset and +running a learner, but rather an iterative process of running the learner, +analyzing the results, modifying the +data and/or the learner, and repeating. Learning is often the quickest +part of this, but that is because we +have already mastered it pretty well! +Feature engineering is more difficult because it is domain-specific, +while learners can be largely general +purpose. However, there is no sharp +frontier between the two, and this is +another reason the most useful learners are those that facilitate incorporating knowledge. +Of course, one of the holy grails +of machine learning is to automate +more and more of the feature engineering process. One way this is often +done today is by automatically generating large numbers of candidate features and selecting the best by (say) +their information gain with respect +to the class. But bear in mind that +features that look irrelevant in isolation may be relevant in combination. +For example, if the class is an XOR of +k input features, each of them by itself carries no information about the +class. (If you want to annoy machine +learners, bring up XOR.) On the other +hand, running a learner with a very +large number of features to find out +which ones are useful in combination +may be too time-consuming, or cause +overfitting. So there is ultimately no +replacement for the smarts you put +into feature engineering. +More Data Beats +a Cleverer Algorithm +Suppose you have constructed the +best set of features you can, but the +classifiers you receive are still not accurate enough. What can you do now? +There are two main choices: design a +better learning algorithm, or gather +more data (more examples, and possibly more raw features, subject to +the curse of dimensionality). Machine +learning researchers are mainly concerned with the former, but pragmatically the quickest path to success is + +review articles +often to just get more data. As a rule +of thumb, a dumb algorithm with lots +and lots of data beats a clever one with +modest amounts of it. (After all, machine learning is all about letting data +do the heavy lifting.) +This does bring up another problem, however: scalability. In most of +computer science, the two main limited resources are time and memory. +In machine learning, there is a third +one: training data. Which one is the +bottleneck has changed from decade +to decade. In the 1980s it tended to +be data. Today it is often time. Enormous mountains of data are available, but there is not enough time +to process it, so it goes unused. This +leads to a paradox: even though in +principle more data means that more +complex classifiers can be learned, in +practice simpler classifiers wind up +being used, because complex ones +take too long to learn. Part of the answer is to come up with fast ways to +learn complex classifiers, and indeed +there has been remarkable progress +in this direction (for example, Hulten +and Domingos11). +Part of the reason using cleverer +algorithms has a smaller payoff than +you might expect is that, to a first approximation, they all do the same. +This is surprising when you consider +representations as different as, say, +sets of rules and neural networks. But +in fact propositional rules are readily +encoded as neural networks, and similar relationships hold between other +representations. All learners essentially work by grouping nearby examples into the same class; the key difference is in the meaning of “nearby.” +With nonuniformly distributed data, +learners can produce widely different +frontiers while still making the same +predictions in the regions that matter +(those with a substantial number of +training examples, and therefore also +where most test examples are likely to +appear). This also helps explain why +powerful learners can be unstable but +still accurate. Figure 3 illustrates this +in 2D; the effect is much stronger in +high dimensions. +As a rule, it pays to try the simplest +learners first (for example, naïve Bayes +before logistic regression, k-nearest +neighbor before support vector machines). More sophisticated learn- + +ers are seductive, but they are usually +harder to use, because they have more +knobs you need to turn to get good results, and because their internals are +more opaque. +Learners can be divided into two +major types: those whose representation has a fixed size, like linear classifiers, and those whose representation +can grow with the data, like decision +trees. (The latter are sometimes called +nonparametric learners, but this is +somewhat unfortunate, since they +usually wind up learning many more +parameters than parametric ones.) +Fixed-size learners can only take advantage of so much data. (Notice how +the accuracy of naive Bayes asymptotes +at around 70% in Figure 2.) Variablesize learners can in principle learn any +function given sufficient data, but in +practice they may not, because of limitations of the algorithm (for example, +greedy search falls into local optima) +or computational cost. Also, because +of the curse of dimensionality, no existing amount of data may be enough. +For these reasons, clever algorithms— +those that make the most of the data +and computing resources available— +often pay off in the end, provided you +are willing to put in the effort. There +is no sharp frontier between designing learners and learning classifiers; +rather, any given piece of knowledge +could be encoded in the learner or +learned from data. So machine learning projects often wind up having a +significant component of learner design, and practitioners need to have +some expertise in it.12 +In the end, the biggest bottleneck +is not data or CPU cycles, but human +Figure 3. Very different frontiers can yield +similar predictions. (+ and – are training +examples of two classes.) + +N. Bayes + +SVM + +kNN + +D. Tree + +cycles. In research papers, learners +are typically compared on measures +of accuracy and computational cost. +But human effort saved and insight +gained, although harder to measure, +are often more important. This favors +learners that produce human-understandable output (for example, rule +sets). And the organizations that make +the most of machine learning are +those that have in place an infrastructure that makes experimenting with +many different learners, data sources, +and learning problems easy and efficient, and where there is a close collaboration between machine learning +experts and application domain ones. +Learn Many Models, Not Just One +In the early days of machine learning, everyone had a favorite learner, +together with some a priori reasons +to believe in its superiority. Most effort went into trying many variations +of it and selecting the best one. Then +systematic empirical comparisons +showed that the best learner varies +from application to application, and +systems containing many different +learners started to appear. Effort now +went into trying many variations of +many learners, and still selecting just +the best one. But then researchers +noticed that, if instead of selecting +the best variation found, we combine +many variations, the results are better—often much better—and at little +extra effort for the user. +Creating such model ensembles is +now standard.1 In the simplest technique, called bagging, we simply generate random variations of the training set by resampling, learn a classifier +on each, and combine the results by +voting. This works because it greatly +reduces variance while only slightly +increasing bias. In boosting, training +examples have weights, and these are +varied so that each new classifier focuses on the examples the previous +ones tended to get wrong. In stacking, +the outputs of individual classifiers +become the inputs of a “higher-level” +learner that figures out how best to +combine them. +Many other techniques exist, and +the trend is toward larger and larger +ensembles. In the Netflix prize, teams +from all over the world competed to +build the best video recommender + +o cto b e r 2 0 1 2 | vo l . 55 | n o. 1 0 | c om m u n icat ion s of t he ac m + +85 + +review articles +system (http://netflixprize.com). As +the competition progressed, teams +found they obtained the best results +by combining their learners with other teams’, and merged into larger and +larger teams. The winner and runnerup were both stacked ensembles of +over 100 learners, and combining the +two ensembles further improved the +results. Doubtless we will see even +larger ones in the future. +Model ensembles should not be +confused with Bayesian model averaging (BMA)—the theoretically +optimal approach to learning.4 In +BMA, predictions on new examples +are made by averaging the individual +predictions of all classifiers in the +hypothesis space, weighted by how +well the classifiers explain the training data and how much we believe +in them a priori. Despite their superficial similarities, ensembles and +BMA are very different. Ensembles +change the hypothesis space (for example, from single decision trees to +linear combinations of them), and +can take a wide variety of forms. BMA +assigns weights to the hypotheses in +the original space according to a fixed +formula. BMA weights are extremely +different from those produced by +(say) bagging or boosting: the latter +are fairly even, while the former are +extremely skewed, to the point where +the single highest-weight classifier +usually dominates, making BMA effectively equivalent to just selecting +it.8 A practical consequence of this is +that, while model ensembles are a key +part of the machine learning toolkit, +BMA is seldom worth the trouble. + +Just because +a function can +be represented +does not mean +it can be learned. + +Simplicity Does Not +Imply Accuracy +Occam’s razor famously states that +entities should not be multiplied beyond necessity. In machine learning, +this is often taken to mean that, given +two classifiers with the same training +error, the simpler of the two will likely +have the lowest test error. Purported +proofs of this claim appear regularly +in the literature, but in fact there are +many counterexamples to it, and the +“no free lunch” theorems imply it cannot be true. +We saw one counterexample previously: model ensembles. The generalization error of a boosted ensemble +86 + +communicatio ns o f th e acm + +| o cto ber 201 2 | vo l . 5 5 | no. 1 0 + +continues to improve by adding classifiers even after the training error has +reached zero. Another counterexample is support vector machines, which +can effectively have an infinite number of parameters without overfitting. +Conversely, the function sign(sin(ax)) +can discriminate an arbitrarily large, +arbitrarily labeled set of points on the +x axis, even though it has only one parameter.23 Thus, contrary to intuition, +there is no necessary connection between the number of parameters of a +model and its tendency to overfit. +A more sophisticated view instead +equates complexity with the size of +the hypothesis space, on the basis that +smaller spaces allow hypotheses to be +represented by shorter codes. Bounds +like the one in the section on theoretical guarantees might then be viewed +as implying that shorter hypotheses +generalize better. This can be further +refined by assigning shorter codes to +the hypotheses in the space we have +some a priori preference for. But +viewing this as “proof” of a trade-off +between accuracy and simplicity is +circular reasoning: we made the hypotheses we prefer simpler by design, +and if they are accurate it is because +our preferences are accurate, not because the hypotheses are “simple” in +the representation we chose. +A further complication arises from +the fact that few learners search their +hypothesis space exhaustively. A +learner with a larger hypothesis space +that tries fewer hypotheses from it +is less likely to overfit than one that +tries more hypotheses from a smaller +space. As Pearl18 points out, the size of +the hypothesis space is only a rough +guide to what really matters for relating training and test error: the procedure by which a hypothesis is chosen. +Domingos7 surveys the main arguments and evidence on the issue of +Occam’s razor in machine learning. +The conclusion is that simpler hypotheses should be preferred because +simplicity is a virtue in its own right, +not because of a hypothetical connection with accuracy. This is probably +what Occam meant in the first place. +Representable Does Not +Imply Learnable +Essentially all representations used in +variable-size learners have associated + +review articles +theorems of the form “Every function +can be represented, or approximated +arbitrarily closely, using this representation.” Reassured by this, fans of +the representation often proceed to +ignore all others. However, just because a function can be represented +does not mean it can be learned. For +example, standard decision tree learners cannot learn trees with more leaves +than there are training examples. In +continuous spaces, representing even +simple functions using a fixed set of +primitives often requires an infinite +number of components. Further, if +the hypothesis space has many local +optima of the evaluation function, as +is often the case, the learner may not +find the true function even if it is representable. Given finite data, time and +memory, standard learners can learn +only a tiny subset of all possible functions, and these subsets are different +for learners with different representations. Therefore the key question is +not “Can it be represented?” to which +the answer is often trivial, but “Can it +be learned?” And it pays to try different +learners (and possibly combine them). +Some representations are exponentially more compact than others for +some functions. As a result, they may +also require exponentially less data to +learn those functions. Many learners +work by forming linear combinations +of simple basis functions. For example, support vector machines form +combinations of kernels centered at +some of the training examples (the +support vectors). Representing parity +of n bits in this way requires 2n basis +functions. But using a representation +with more layers (that is, more steps +between input and output), parity can +be encoded in a linear-size classifier. +Finding methods to learn these deeper +representations is one of the major research frontiers in machine learning.2 +Correlation Does Not +Imply Causation +The point that correlation does not +imply causation is made so often that +it is perhaps not worth belaboring. +But, even though learners of the kind +we have been discussing can only +learn correlations, their results are +often treated as representing causal +relations. Isn’t this wrong? If so, then +why do people do it? + +More often than not, the goal +of learning predictive models is to +use them as guides to action. If we +find that beer and diapers are often +bought together at the supermarket, then perhaps putting beer next +to the diaper section will increase +sales. (This is a famous example in +the world of data mining.) But short +of actually doing the experiment it is +difficult to tell. Machine learning is +usually applied to observational data, +where the predictive variables are not +under the control of the learner, as +opposed to experimental data, where +they are. Some learning algorithms +can potentially extract causal information from observational data, but +their applicability is rather restricted.19 On the other hand, correlation +is a sign of a potential causal connection, and we can use it as a guide to +further investigation (for example, +trying to understand what the causal +chain might be). +Many researchers believe that causality is only a convenient fiction. For +example, there is no notion of causality in physical laws. Whether or not +causality really exists is a deep philosophical question with no definitive +answer in sight, but there are two +practical points for machine learners. First, whether or not we call them +“causal,” we would like to predict the +effects of our actions, not just correlations between observable variables. +Second, if you can obtain experimental data (for example by randomly assigning visitors to different versions of +a Web site), then by all means do so.14 +Conclusion +Like any discipline, machine learning has a lot of “folk wisdom” that can +be difficult to come by, but is crucial +for success. This article summarized +some of the most salient items. Of +course, it is only a complement to the +more conventional study of machine +learning. Check out http://www. +cs.washington.edu/homes/pedrod/ +class for a complete online machine +learning course that combines formal +and informal aspects. There is also a +treasure trove of machine learning +lectures at http://www.videolectures. +net. A good open source machine +learning toolkit is Weka.24 +Happy learning! + +References +1. Bauer, E. and Kohavi, R. An empirical comparison of +voting classification algorithms: Bagging, boosting +and variants. Machine Learning 36 (1999), 105–142. +2. Bengio, Y. Learning deep architectures for AI. +Foundations and Trends in Machine Learning 2, 1 +(2009), 1–127. +3. Benjamini, Y. and Hochberg, Y. Controlling the false +discovery rate: A practical and powerful approach +to multiple testing. Journal of the Royal Statistical +Society, Series B, 57 (1995), 289–300. +4. Bernardo, J.M. and Smith, A.F.M. Bayesian Theory. +Wiley, NY, 1994. +5. Blumer, A., Ehrenfeucht, A., Haussler, D. and +Warmuth, M.K. Occam’s razor. Information +Processing Letters 24 (1987), 377–380. +6. Cohen, W.W. Grammatically biased learning: +Learning logic programs using an explicit antecedent +description language. Artificial Intelligence 68 +(1994), 303–366. +7. Domingos, P. The role of Occam’s razor in knowledge +discovery. Data Mining and Knowledge Discovery 3 +(1999), 409–425. +8. Domingos, P. Bayesian averaging of classifiers and +the overfitting problem. In Proceedings of the 17th +International Conference on Machine Learning +(Stanford, CA, 2000), Morgan Kaufmann, San Mateo, +CA, 223–230. +9. Domingos, P. A unified bias-variance decomposition +and its applications. In Proceedings of the 17th +International Conference on Machine Learning +(Stanford, CA, 2000), Morgan Kaufmann, San Mateo, +CA, 231–238. +10. Domingos, P. and Pazzani, M. On the optimality of +the simple Bayesian classifier under zero-one loss. +Machine Learning 29 (1997), 103–130. +11. Hulten, G. and Domingos, P. Mining complex models +from arbitrarily large databases in constant time. In +Proceedings of the 8th ACM SIGKDD International +Conference on Knowledge Discovery and Data Mining +(Edmonton, Canada, 2002). ACM Press, NY, 525–531. +12. Kibler, D. and Langley, P. Machine learning as an +experimental science. In Proceedings of the 3rd +European Working Session on Learning (London, UK, +1988). Pitman. +13. Klockars, A.J. and Sax, G. Multiple Comparisons. +Sage, Beverly Hills, CA, 1986. +14. Kohavi, R., Longbotham, R., Sommerfield, D. and +Henne, R. Controlled experiments on the Web: +Survey and practical guide. Data Mining and +Knowledge Discovery 18 (2009), 140–181. +15. Manyika, J., Chui, M., Brown, B., Bughin, J., Dobbs, +R., Roxburgh, C. and Byers, A. Big data: The next +frontier for innovation, competition, and productivity. +Technical report, McKinsey Global Institute, 2011. +16. Mitchell, T.M. Machine Learning. McGraw-Hill, +NY, 1997. +17. Ng, A.Y. Preventing “overfitting” of cross-validation +data. In Proceedings of the 14th International +Conference on Machine Learning (Nashville, TN, +1997). Morgan Kaufmann, San Mateo, CA, 245–253. +18. Pearl, J. On the connection between the complexity +and credibility of inferred models. International +Journal of General Systems 4 (1978), 255–264. +19. Pearl, J. Causality: Models, Reasoning, and +Inference. Cambridge University Press, Cambridge, +UK, 2000. +20. Quinlan, J.R. C4.5: Programs for Machine Learning. +Morgan Kaufmann, San Mateo, CA, 1993. +21. Richardson, M. and P. Domingos. Markov logic +networks. Machine Learning 62 (2006), 107–136. +22. Tenenbaum, J., Silva, V. and Langford, J. A global +geometric framework for nonlinear dimensionality +reduction. Science 290 (2000), 2319–2323. +23. Vapnik, V.N. The Nature of Statistical Learning +Theory. Springer, NY, 1995. +24. Witten, I., Frank, E. and Hall, M. Data Mining: +Practical Machine Learning Tools and Techniques, +3rd Edition. Morgan Kaufmann, San Mateo, CA, 2011. +25. Wolpert, D. The lack of a priori distinctions between +learning algorithms. Neural Computation 8 (1996), +1341–1390. +Pedro Domingos (pedrod@cs.washington.edu) is a +professor in the Department of Computer Science and +Engineering at the University of Washington, Seattle. + +© 2012 ACM 0001-0782/12/10 $15.00 + +o cto b e r 2 0 1 2 | vo l . 55 | n o. 1 0 | c om m u n icat ion s of t he ac m + +87 -> A dumb algorithm with lots and lots of data beats a clever one with modest amounts of it. diff --git a/docs/evidence/lones_2021_ml_pitfalls.md b/docs/evidence/lones_2021_ml_pitfalls.md index 1dc49c6..efa5d34 100644 --- a/docs/evidence/lones_2021_ml_pitfalls.md +++ b/docs/evidence/lones_2021_ml_pitfalls.md @@ -1,30 +1,484 @@ -Source: https://arxiv.org/abs/2108.02497 (v5; updated annually since 2021) -Title: "How to avoid machine learning pitfalls: a guide for academic researchers" — Michael A. Lones (Heriot-Watt University) -Fetched-via: PDF downloaded from arxiv.org, text extracted with pdfplumber, 2026-06-11 -Fetch-status: verbatim excerpts; line breaks rejoined, ligature artifacts fixed +Source: https://arxiv.org/abs/2108.02497 (v5) +Title: "How to avoid machine learning pitfalls: a guide for academic researchers" -- Michael A. Lones (Heriot-Watt University) +Fetched-via: curl https://r.jina.ai/https://arxiv.org/pdf/2108.02497, 2026-08-14 +Fetch-status: verbatim, full paper (33 pages) -# How to avoid machine learning pitfalls (excerpts) +Title: 2108.02497v5.pdf -Abstract: +URL Source: https://arxiv.org/pdf/2108.02497 -> Mistakes in machine learning practice are commonplace, and can result in a loss of confidence in the findings and products of machine learning. This guide outlines common mistakes that occur when using machine learning, and what can be done to avoid them. Whilst it should be accessible to anyone with a basic understanding of machine learning techniques, it focuses on issues that are of particular concern within academic research, such as the need to do rigorous comparisons and reach valid conclusions. It covers five stages of the machine learning process: what to do before model building, how to reliably build models, how to robustly evaluate models, how to compare models fairly, and how to report results. +Published Time: Fri, 30 Aug 2024 01:06:00 GMT -Structure (from the introduction): +Number of Pages: 33 -> The review is divided into five sections. *Before you start to build models* covers issues that can occur early in the ML process, and focuses on the correct use of data and adequate consideration of the context in which ML is being applied. *How to reliably build models* then covers pitfalls that occur during the selection and training of models and their components. *How to robustly evaluate models* presents pitfalls that can lead to an incorrect understanding of model performance. *How to compare models fairly* then extends this to the situation where models are being compared, discussing how common pitfalls can lead to misleading findings. *How to report your results* focuses on reproducibility and factors that can lead to incomplete or deceptive reporting. +Markdown Content: +# How to avoid machine learning pitfalls: a guide for academic researchers -The full do/don't list (table of contents, v5) — this is the exhaustive-checklist value of the paper: +# Michael A. Lones ∗ -> 2.1 Do think about how and where you will use data / 2.2 Do take the time to understand your data / 2.3 Don't look at all your data / 2.4 Do clean your data / 2.5 Do make sure you have enough data / 2.6 Do talk to domain experts / 2.7 Do survey the literature / 2.8 Do think about how your model will be deployed -> 3.1 Don't allow test data to leak into the training process / 3.2 Do try out a range of different models / 3.3 Don't use inappropriate models / 3.4 Do keep up with progress in deep learning (and its pitfalls) / 3.5 Don't assume deep learning will be the best approach / 3.6 Do be careful where and how you do feature selection / 3.7 Do optimise your model's hyperparameters / 3.8 Do avoid learning spurious correlations -> 4.1 Do use an appropriate test set / 4.2 Don't do data augmentation before splitting your data / 4.3 Do avoid sequential overfitting / 4.4 Do evaluate a model multiple times / 4.5 Do save some data to evaluate your final model instance / 4.6 Do choose metrics carefully / 4.7 Do consider model fairness / 4.8 Don't ignore temporal dependencies in time series data -> 5.1 Don't assume a bigger number means a better model / 5.2 Do use meaningful baselines / 5.3 Do use statistical tests when comparing models / 5.4 Do correct for multiple comparisons / 5.5 Don't always believe results from community benchmarks / 5.6 Do combine models (carefully) -> 6.1 Do be transparent / 6.2 Do report performance in multiple ways / 6.3 Don't generalise beyond the data / 6.4 Do be careful when reporting statistical significance / 6.5 Do look at your models / 6.6 Do use a machine learning checklist +Abstract -Section 3.1, "Don't allow test data to leak into the training process": +Mistakes in machine learning practice are commonplace, and can result in a loss of confidence in the findings and products of machine learning. This guide outlines common mistakes that occur when using machine learning, and what can be done to avoid them. Whilst it should be accessible to anyone with a basic understanding of machine learning techniques, it focuses on issues that are of particular concern within academic research, such as the need to do rigorous comparisons and reach valid conclusions. It covers five stages of the machine learning process: what to do before model building, how to reliably build models, how to robustly evaluate models, how to compare models fairly, and how to report results. -> A common problem is allowing information about this data to leak into the configuration, training or selection of models. When this happens, the data no longer provides a reliable measure of generality, and this is a common reason why published ML models often fail to generalise to real world data. There are a number of ways that information can leak from a test set. Some of these seem quite innocuous. For instance, during data preparation, using information about the means and ranges of variables within the whole data set to carry out variable scaling or imputation — in order to prevent information leakage, these statistics should be calculated using only the training data. [...] The best thing you can do to prevent these issues is to partition off a subset of your data right at the start of your project, and only use this independent test set once to measure the generality of a single model at the end. +# 1 Introduction -Section 4.8, "Don't ignore temporal dependencies in time series data": +It’s easy to make mistakes when applying machine learning (ML), and these mistakes can result in ML models that fail to work as expected when applied to data not seen during training and testing [Liao et al., 2021]. This is a problem for practitioners, since it leads to the failure of ML projects. However, it is also a problem for society, since it erodes trust in the findings and products of ML [Gibney, 2022]. This guide aims to help newcomers avoid some of these mistakes. It’s written by an academic, and focuses on lessons learnt whilst doing ML research in academia. Whilst primarily aimed at students and scientific researchers, it should be accessible to anyone getting started in ML, and only assumes a basic knowledge of ML techniques. However, unlike similar guides aimed at a more general audience, it includes topics that are of a particular concern to academia, such as the need to rigorously evaluate and compare models in order to get work published. To make it more readable, the guidance is written informally, in a Dos and Don’ts style. It’s not intended to be exhaustive, and references (with publicly-accessible URLs where available) are provided for further reading. Since it doesn’t cover issues specific to partic-ular academic subjects, it’s recommended that readers also consult subject-specific guid-ance where available, e.g. in clinical medicine [Stevens et al., 2020], genomics [Whalen et al., 2022], environmental research [Zhu et al., 2023], materials science [Karande et al., -> Most notably, time series data are subject to a particular kind of data leakage known as look ahead bias. This occurs when some or all of the data points used to train the model occur later in the time series than those used to test the model. In effect, this can allow knowledge of the future to leak into training, and this can then bias the test performance. A situation where this commonly occurs is when standard cross-validation is applied to time series data, since it results in the training folds in all but one of the cross-validation iterations containing data that is in the future relative to the test fold. +> ∗ + +School of Mathematical and Computer Sciences, Heriot-Watt University, Edinburgh, Scotland, UK, Email: m.lones@hw.ac.uk, Web: www.macs.hw.ac.uk/ ∼ml355, Substack: Fetch Decode Execute. + +1 + +> arXiv:2108.02497v5 [cs.LG] 29 Aug 2024 + +2022], business and marketing [Van Giffen et al., 2022], computer security [Arp et al., 2022] and social science [Malik, 2020]. The review is divided into five sections. Before you start to build models covers issues that can occur early in the ML process, and focuses on the correct use of data and adequate consideration of the context in which ML is being applied. How to reliably build models then covers pitfalls that occur during the selection and training of models and their components. How to robustly evaluate models presents pitfalls that can lead to an incorrect understanding of model performance. How to compare models fairly then extends this to the situation where models are being compared, discussing how common pitfalls can lead to misleading findings. How to report your results focuses on reproducibility and factors that can lead to incomplete or deceptive reporting. + +# Changes + +ML pitfalls are not static, and continue to evolve as ML develops. To address this, this guide has been updated annually since it was first released in 2021, and it will continue to be updated in the future. Feedback is welcome. If you cite it, please include the arXiv version number (currently v5 1). + +Changes from v4 Added Do use meaningful baselines, Do clean your data and Do consider model fairness. Extended Do look at your models, Do make sure you have enough data, Do think about how your model will be deployed, Don’t allow test data to leak into the training process and Do evaluate a model multiple times. + +Changes from v3 Added Do use a machine learning checklist and Do think about how and where you will use data. Rewrote Do evaluate a model multiple times. Revised Do keep up with progress in deep learning (and its pitfalls), Do be careful where and how you do feature selection, Do avoid sequential overfitting, Do choose metrics carefully and Do combine models (carefully). Extended Do use an appropriate test set. + +Changes from v2 Added illustrations. Added Do avoid learning spurious correlations, Don’t ignore temporal dependencies in time series data and Do keep up with progress in deep learning (and its pitfalls). + +Changes from v1 Added Don’t do data augmentation before splitting your data and Don’t assume deep learning will be the best approach. Rewrote Don’t use inappropriate models. Expanded Don’t allow test data to leak into the training process, Do be careful when reporting statistical significance and Do be transparent. + +> 1This version is published as “Avoiding machine learning pitfalls” in Patterns (Cell Press) [Lones, 2024] + +2Contents + +1 Introduction 1 + +2 Before you start to build models 4 + +2.1 Do think about how and where you will use data . . . . . . . . . . . . . . . . . 4 + +2.2 Do take the time to understand your data . . . . . . . . . . . . . . . . . . . . 4 + +2.3 Don’t look at all your data . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 + +2.4 Do clean your data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 + +2.5 Do make sure you have enough data . . . . . . . . . . . . . . . . . . . . . . . 6 + +2.6 Do talk to domain experts . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 + +2.7 Do survey the literature . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 + +2.8 Do think about how your model will be deployed . . . . . . . . . . . . . . . . . 7 + +3 How to reliably build models 7 + +3.1 Don’t allow test data to leak into the training process . . . . . . . . . . . . . . 8 + +3.2 Do try out a range of different models . . . . . . . . . . . . . . . . . . . . . . 9 + +3.3 Don’t use inappropriate models . . . . . . . . . . . . . . . . . . . . . . . . . . 9 + +3.4 Do keep up with progress in deep learning (and its pitfalls) . . . . . . . . . . . . 9 + +3.5 Don’t assume deep learning will be the best approach . . . . . . . . . . . . . . 10 + +3.6 Do be careful where and how you do feature selection . . . . . . . . . . . . . . 11 + +3.7 Do optimise your model’s hyperparameters . . . . . . . . . . . . . . . . . . . . 11 + +3.8 Do avoid learning spurious correlations . . . . . . . . . . . . . . . . . . . . . . 13 + +4 How to robustly evaluate models 14 + +4.1 Do use an appropriate test set . . . . . . . . . . . . . . . . . . . . . . . . . . 14 + +4.2 Don’t do data augmentation before splitting your data . . . . . . . . . . . . . . 15 + +4.3 Do avoid sequential overfitting . . . . . . . . . . . . . . . . . . . . . . . . . . 15 + +4.4 Do evaluate a model multiple times . . . . . . . . . . . . . . . . . . . . . . . . 15 + +4.5 Do save some data to evaluate your final model instance . . . . . . . . . . . . . 17 + +4.6 Do choose metrics carefully . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 + +4.7 Do consider model fairness . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 + +4.8 Don’t ignore temporal dependencies in time series data . . . . . . . . . . . . . . 19 + +5 How to compare models fairly 19 + +5.1 Don’t assume a bigger number means a better model . . . . . . . . . . . . . . . 19 + +5.2 Do use meaningful baselines . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 + +5.3 Do use statistical tests when comparing models . . . . . . . . . . . . . . . . . . 21 + +5.4 Do correct for multiple comparisons . . . . . . . . . . . . . . . . . . . . . . . . 22 + +5.5 Don’t always believe results from community benchmarks . . . . . . . . . . . . 22 + +5.6 Do combine models (carefully) . . . . . . . . . . . . . . . . . . . . . . . . . . 23 + +6 How to report your results 23 + +6.1 Do be transparent . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 + +6.2 Do report performance in multiple ways . . . . . . . . . . . . . . . . . . . . . 24 + +6.3 Don’t generalise beyond the data . . . . . . . . . . . . . . . . . . . . . . . . . 24 + +6.4 Do be careful when reporting statistical significance . . . . . . . . . . . . . . . 25 + +6.5 Do look at your models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 + +6.6 Do use a machine learning checklist . . . . . . . . . . . . . . . . . . . . . . . . 26 + +7 Final thoughts 27 + +32 Before you start to build models + +It’s normal to want to rush into training and evaluating models, but it’s important to take the time to think about the goals of a project, to fully understand the data that will be used to support these goals, to consider any limitations of the data that need to be addressed, and to understand what’s already been done in your field. If you don’t do these things, then you may end up with results that are hard to publish, or models that are not appropriate for their intended purpose. + +2.1 Do think about how and where you will use data + +Data is central to most ML projects, but is often in short supply. Therefore it’s impor-tant to think carefully about what data you need and how and where you will use it. Abstractly, you need data for two things, training models and testing models. However, for various reasons, this does not necessarily translate into using a single dataset divided into two parts. To begin with, model development often involves a period of experimen-tation: trying out different models with different hyperparameters, and preprocessing the data in different ways. To avoid overfitting (see Do avoid sequential overfitting), this process requires a separate validation set, i.e. an additional set of training data that’s not used directly in training or testing models. If you have no prior idea of what modelling approach you’re going to use, then this experimentation phase could potentially involve a lot of comparisons. Due to the multiplicity effect (see Do correct for multiple compar-isons), the more comparisons you do, the more likely you are to overfit the validation data, and so the less useful the validation set will become in guiding your modelling decisions. So, in practice you might want to set aside multiple validation sets for this. Then there’s the question of how you adequately test your selected model. Because it has the same biases as the training data, a test set taken from the same dataset as the training data may not be sufficient to measure the model’s generality — see Do use an appropriate test set and Do report performance in multiple ways for more on this — meaning that, in practice, you may need more than one test dataset to robustly evaluate your model. Also be aware that you will often need additional test data when using cross-validation; see Do save some data to evaluate your final model instance. + +2.2 Do take the time to understand your data + +Eventually you will want to publish your work. This is a lot easier to do if your data is from a reliable source, has been collected using a reliable methodology, and is of good quality. For instance, if you are using data collected from an internet resource, make sure you know where it came from. Is it described in a paper? If so, take a look at the paper; make sure it was published somewhere reputable, and check whether the authors mention any limitations of the data. Do not assume that, because a data set has been used by a number of papers, it is of good quality — sometimes data is used just because it is easy to get hold of, and some widely used data sets are known to have significant limitations (see Paullada et al. [2021] for a discussion of this). If you 4train your model using bad data, then you will most likely generate a bad model: aprocess known as garbage in garbage out . One way to avoid bad data sets is to build a direct relationship with people who generate data, since this increases the likelihood of obtaining a good-quality dataset that meets your needs. It also avoids problems of overfitting community benchmarks; see Don’t always believe results from community benchmarks. Yet regardless of where your data comes from, always begin by making sure that your data makes sense. Do some exploratory data analysis (see Cox [2017] for suggestions). Look for missing or inconsistent records. It is much easier to do this now, before you train a model, rather than later, when you’re trying to explain to reviewers why you used bad data. + +2.3 Don’t look at all your data + +As you look at data, it is quite likely that you will spot patterns and make insights that guide your modelling. This is another good reason to look at data. However, it is important that you do not make untestable assumptions that will later feed into your model. The “untestable” bit is important here; it’s fine to make assumptions, but these should only feed into the training of the model, not the testing. So, to ensure this is the case, you should avoid looking closely at any test data in the initial exploratory analysis stage. Otherwise you might, consciously or unconsciously, make assumptions that limit the generality of your model in an untestable way. This is a theme I will return to several times, since the leakage of information from the test set into the training process is a common reason why ML models fail to generalise. See Don’t allow test data to leak into the training process for more on this. + +2.4 Do clean your data + +Even good-quality datasets will have issues. Some of these come from unavoidable noise or omissions in the data collection process, others are due to human error during collection or collation. Whatever the cause, it’s important to identify any issues, and do this before you start to build models. One common problem to look out for is data duplication, i.e. the unintentional inclusion of multiple copies of a data point. This can cause serious problems when a model is evaluated (see Don’t do data augmentation + +before splitting your data for an example), so should be identified and removed early on. Another common problem is missing values. Some models can cope with these, but many can’t, and so you’ll have to replace missing values with something else before they can be trained. There are various forms of imputation that can be used to achieve this; see Emmanuel et al. [2021] for a review. If you do imputation, be careful to avoid data leaks during imputation — see Don’t allow test data to leak into the training process. You should also check for outliers in your data, but only remove these if they are likely to be the result of noise or error rather than being natural extremes of the underlying data-generating process. For example, if a person’s age is greater than 150, then it’s probably an error; if it’s 110, then it could be a natural outlier. A related issue is meaningless or inconsistent data, for instance a person with a negative age. Data cleaning can be 5a time-consuming process, and becomes more challenging as the complexity of data increases. For this reason, many people have explored automating data cleaning using ML approaches; see Cˆ ot´ e et al. [2024] for a review. + +2.5 Do make sure you have enough data + +If you don’t have enough data, then it may not be possible to train a model that gener-alises. Working out whether this is the case can be challenging, and may not be evident until you start building models: it all depends on the signal to noise ratio in the data set. If the signal is strong, then you can get away with less data; if it’s weak, then you need more data. If you can’t get more data — and this is a common issue in many research fields — then you can try using data augmentation techniques (see Wang et al. [2024], and for time series data, Iglesias et al. [2023]). These can be quite effective for boosting small data sets, though Don’t do data augmentation before splitting your data. Data augmentation is also useful in situations where you have limited data in certain parts of your data set, e.g. in classification problems where you have less samples in some classes than others, a situation known as class imbalance . See Haixiang et al. [2017] for a review of methods for dealing with this; also see Do choose metrics carefully. Another option for dealing with small data sets is to use transfer learning — see Do keep up with progress in deep learning (and its pitfalls). A danger when using small datasets is that different data partitions may be biased, for instance in terms of the quality or difficulty of data they contain. For this reason, it is advisable to consider frequent repartitioning. Cross-validation (see Do evaluate a model multiple times) is an efficient way of achieving this in small data sets. If you have limited data, then it’s also likely that you will have to limit the complexity of the ML models you use, since models with many parameters, like deep neural networks, can easily overfit small data sets (see Don’t assume deep learning will be the best approach). Regardless of how you handle the problem of limited data, it’s important to identify this issue early on, and come up with a suitable strategy to mitigate against it. + +2.6 Do talk to domain experts + +Domain experts can be very valuable. They can help you to understand which problems are useful to solve, they can help you choose the most appropriate feature set and ML model to use, and they can help you publish to the most appropriate audience. Failing to consider the opinion of domain experts can lead to projects which don’t solve useful problems, or which solve useful problems in inappropriate ways. An example of the latter is using an opaque ML model to solve a problem where there is a strong need to understand how the model reaches an outcome, e.g. in making medical or financial decisions (see Rudin [2019]). At the beginning of a project, domain experts can help you to understand the data, and point you towards features that are likely to be predictive. At the end of a project, they can help you to publish in domain-specific journals, and hence reach an audience that is most likely to benefit from your research. 62.7 Do survey the literature + +You’re probably not the first person to throw ML at a particular problem domain, so it’s important to understand what has and hasn’t been done previously. Other people having worked on the same problem isn’t a bad thing; academic progress is typically an iterative process, with each study providing information that can guide the next. It may be discouraging to find that someone has already explored your great idea, but they most likely left plenty of avenues of investigation still open, and their previous work can be used as justification for your work. To ignore previous studies is to potentially miss out on valuable information. For example, someone may have tried your proposed approach before and found fundamental reasons why it won’t work (and therefore saved you a few years of frustration), or they may have partially solved the problem in a way that you can build on. So, it’s important to do a literature review before you start work; leaving it too late may mean that you are left scrambling to explain why you are covering the same ground or not building on existing knowledge when you come to write a paper. + +2.8 Do think about how your model will be deployed + +Why do you want to build an ML model? This is an important question, and the answer should influence the process you use to develop your model. Many academic studies are just that — studies — and not really intended to produce models that will be used in the real world. This is fair enough, since the process of building and analysing models can itself give very useful insights into a problem. However, for many academic studies, the eventual goal is to produce an ML model that can be deployed in a real world situation. If this is the case, then it’s worth thinking early on about how it is going to be deployed. For instance, if it’s going to be deployed in a resource-limited environment, such as a sensor or a robot, this may place limitations on the complexity of the model. If there are time constraints, e.g. a classification of a signal is required within milliseconds, then this also needs to be taken into account when selecting a model. If using deep learning, then energy costs and carbon footprint may be a consideration, and if using LLMs, there may be further operational costs for hosting or accessing foundation models. Another consideration is how the model is going to be tied into the broader software system within which it is deployed; this procedure is often far from simple (see Sculley et al. [2015]). However, emerging approaches such as ML Ops aim to address some of the difficulties; see Kreuzberger et al. [2023] for a review, and Shankar et al. [2022] for a discussion of common challenges when operationalising ML models. + +# 3 How to reliably build models + +Building models is one of the more enjoyable parts of ML. With modern ML frameworks, it’s easy to throw all manner of approaches at your data and see what sticks. However, this can lead to a disorganised mess of experiments that’s hard to justify and hard to write up. So, it’s important to approach model building in an organised manner, making sure you use data correctly, and putting adequate consideration into the choice of models. 7Training set Model training + +> Predictive model +> (classifier or regression) +> Test set +> Estimated Performance Training set Model training +> Predictive model +> (classifier or regression) +> Test set +> Over-Estimated Performance + +Figure 1: See Don’t allow test data to leak into the training process. [left] How things should be, with the training set used to train the model, and the test set used to measure its generality. [right] When there’s a data leak, the test set can implicitly become part of the training process, meaning that it no longer provides a reliable measure of generality. + +3.1 Don’t allow test data to leak into the training process + +It’s essential to have data that you can use to measure how well your model generalises. A common problem is allowing information about this data to leak into the configuration, training or selection of models (see Figure 1). When this happens, the data no longer provides a reliable measure of generality, and this is a common reason why published ML models often fail to generalise to real world data. There are a number of ways that information can leak from a test set. Some of these seem quite innocuous. For instance, during data preparation, using information about the means and ranges of variables within the whole data set to carry out variable scaling or imputation — in order to prevent information leakage, these statistics should be calculated using only the training data. Other common examples of information leakage are carrying out feature selection before partitioning the data (see Do be careful where and how you do feature selection), using the same test data to evaluate the generality of multiple models (see Do avoid sequential overfitting and Don’t always believe results from community benchmarks), and applying data augmentation before splitting off the test data (see Don’t do data augmentation before splitting your data). The best thing you can do to prevent these issues is to partition off a subset of your data right at the start of your project, and only use this independent test set once to measure the generality of a single model at the end of the project (see Do save some data to evaluate your final model instance). There are also forms of data leakage which are specific to certain types of data. Time series data is particularly problematic, since the order of samples is significant, and random splits can easily cause leakage and overfitting — see Don’t ignore temporal dependencies in time series data for more on this. Even for non-time series data, the experimental conditions used to generate data sets may lead to temporal dependencies, or other problematic conditions such as duplicated or similar samples — see Do use an appropriate test set for an example. In order to prevent leakage, these kinds of issues need to be identified and taken into account when splitting data. For a broader discussion of data leakage, see Kapoor and Narayanan [2023]. 83.2 Do try out a range of different models + +Generally speaking, there’s no such thing as a single best ML model. In fact, there’s a proof of this, in the form of the No Free Lunch theorem, which shows that no ML approach is any better than any other when considered over every possible problem [Wolpert, 2002]. So, your job is to find the ML model that works well for your particular problem. There is some guidance on this. For example, you can consider the inductive biases of ML models; that is, the kind of relationships they are capable of modelling. For instance, linear models, such as linear regression and logistic regression, are a good choice if you know there are no important non-linear relationships between the features in your data, but a bad choice otherwise. Good quality research on closely related problems may also be able to point you towards models that work particularly well. However, a lot of the time you’re still left with quite a few choices, and the only way to work out which model is best is to try them all. Fortunately, modern ML libraries, such as scikit-learn [Varoquaux et al., 2015] in Python, tidymodels [Kuhn and Wickham, 2020] in R, and MLJ [Blaom et al., 2020] in Julia, allow you to try out multiple models with only small changes to your code, so there’s no reason not to try them all out and find out for yourself which one works best. However, Don’t use inappropriate models, and use a validation set, rather than the test set, to evaluate them (see Do avoid sequential overfitting). When comparing models, Do optimise your model’s hyperparameters and Do evaluate a model multiple times to make sure you’re giving them all a fair chance, and Do correct for multiple comparisons when you publish your results. + +3.3 Don’t use inappropriate models + +By lowering the barrier to implementation, modern ML libraries also make it easy to apply inappropriate models to your data. This, in turn, could look bad when you try to publish your results. A simple example of this is applying models that expect categorical features to a dataset containing numerical features, or vice versa. Some ML libraries allow you to do this, but it may result in a poor model due to loss of information. If you really want to use such a model, then you should transform the features first; there are various ways of doing this, ranging from simple one-hot encodings to complex learned embeddings. Other examples of inappropriate model choice include using a classification model where a regression model would make more sense (or vice versa), attempting to apply a model that assumes no dependencies between variables to time series data, or using a model that is unnecessarily complex (see Don’t assume deep learning will be the best approach). Also, if you’re planning to use your model in practice, Do think about how your model will be deployed, and don’t use models that aren’t appropriate for your use case. + +3.4 Do keep up with progress in deep learning (and its pitfalls) + +Whilst deep learning may not always be the best solution (see Don’t assume deep learning will be the best approach), if you are going to use deep learning, then it’s advisable to try and keep up with recent developments in this fast-moving field. Figure 2 summarises 91950 1960 1970 1980 1990 2000 2010 2020 + +> McCulloch & Pitts neuron Perceptron Backpropagation Hopfield RNN Boltzmann machine Multi-layer Perceptron Jordan RNN Q-learning Autoencoders CNN LSTM ReLU DBN GANs Adam Inception modules Residual blocks Diffusion models Capsule nets Transformers ChatGPT Components Feedforward Recurrent Optimisers GRU SOM Attention Hebbian learning Spiking neuron Batch normalisation NAS Transfer learning Pooling Convolution Dropout + +Figure 2: See Do keep up with progress in deep learning (and its pitfalls). A rough history of neural networks and deep learning, showing what I consider to be the mile-stones in their development. For a far more thorough account of the field’s historical development, take a look at Schmidhuber [2015]. some of the important developments over time. Multilayer perceptrons (MLP) and recurrent neural networks (particularly LSTM) have been around for some time, but have largely been subsumed by newer models such as convolutional neural networks (CNN) [Li et al., 2021] and transformers [Lin et al., 2022]. For example, transformers have become the go-to model for processing sequential data (e.g. natural language), and are increasingly being applied to other data types too, such as images [Khan et al., 2022]. A prominent downside of both transformers and deep CNNs is that they have many parameters and therefore require a lot of data to train them. However, an option for small data sets is to use transfer learning , where a model is pre-trained on a large generic data set and then fine-tuned on the data set of interest [Han et al., 2021]. Larger pre-trained models, many of which are freely shared on websites such as Hugging Face, are known as foundation models ; see Zhou et al. [2023] for a survey. Whilst powerful, these come with their own set of pitfalls. For example, their ability to fully memorise input data is the cause of data security and privacy concerns [Li et al., 2023]. The use of opaque, often poorly documented, training datasets also leads to pitfalls when fitting them into broader ML pipelines (see Do combine models (carefully) for more info) and comparing them fairly with other ML models (see Don’t assume a bigger number means a better model and Don’t always believe results from community benchmarks). For an extensive, yet accessible, guide to deep learning, see Zhang et al. [2023]. + +3.5 Don’t assume deep learning will be the best approach + +A common pitfall is to assume that deep neural networks will provide the best solution to any problem, and consequently fail to try out other, possibly more appropriate, models. 10 Whilst deep learning is great for certain tasks, it is not good at everything; there are plenty of examples of it being out-performed by “old fashioned” machine learning models such as random forests and SVMs. See, for instance, Grinsztajn et al. [2022], who show that tree-based models often outperform deep learners on tabular data. Certain kinds of deep neural network architecture may also be ill-suited to certain kinds of data: see, for example, Zeng et al. [2023], who argue that transformers are not well-suited to time series forecasting. There are also theoretical reasons why any one kind of model won’t always be the best choice (see Do try out a range of different models). In particular, a deep neural network is unlikely to be a good choice if you have limited data, if domain knowledge suggests that the underlying pattern is quite simple, or if the model needs to be interpretable. This last point is particularly worth considering: a deep neural network is essentially a very complex piece of decision making that emerges from interactions between a large number of non-linear functions. Non-linear functions are hard to follow at the best of times, but when you start joining them together, their behaviour gets very complicated very fast. Whilst explainable AI methods (see Do look at your models) can shine some light on the workings of deep neural networks, they can also mislead you by ironing out the true complexities of the decision space (see Molnar et al. [2020]). For this reason, you should take care when using either deep learning or explainable AI for models that are going to make high stakes or safety critical decisions; see Rudin [2019] for more on this. + +3.6 Do be careful where and how you do feature selection + +A common stage of training a model is to carry out feature selection (surveyed by Cai et al. [2018]). When doing this, it is important to treat it as part of model train-ing, and not something more general that you do before model training. A particularly common error is to do feature selection on the whole data set before splitting off the test set, something that will result in information leaking from the test set into the training process (see Don’t allow test data to leak into the training process). Instead, you should only use the training set to select the features which are used in both the training set and the test set (see Figure 3). The same is true when doing dimensional-ity reduction . For example, if you’re using principal component analysis (PCA), the component weightings should be determined by looking only at the training data; the same weightings should then be applied to the test set. Special care should be taken when using autoencoders for dimensionality reduction — see Do combine models (care-fully). If you’re doing cross-validation (see Do evaluate a model multiple times) then it’s important to carry out feature selection or dimensionality reduction independently within each iteration, each time using just the training folds (see Figure 3, bottom). + +3.7 Do optimise your model’s hyperparameters + +Many models have hyperparameters — that is, numbers or settings that affect the configuration of the model. Examples include the kernel function used in an SVM, the number of trees in a random forest, and the architecture of a neural network. Many 11 F8 F7 F6 F5 F4 F3 F2 F1 + +> 258582426742713137771155028621733283623416135365 + +Training set Test set + +Train + +Predictive model + +(classifier or regression) + +> F6 F5 F2 +> 854423775867833136 + +Full data set Feature selection Split data Training set Test set + +Train + +Predictive model + +(classifier or regression) + +2) Select features using training set 1) Split data + +> F8 F7 F6 F5 F4 F3 F2 F1 +> 258582426742713137771155028621733283623416135365 + +Full data set + +> F7 F5 F2 +> 554723775 +> F7 F5 F2 +> 267233636 + +3) Same features used for test set F6 F5 F2 + +> 867833136846625664747853335675824746 +> F6 F5 F2 +> 854423775 + +Cross-validation iteration 1 + +> F7 F6 F5 F2 +> 585474237775784616256664274778538335167558247746 +> F7 F6 F5 F2 +> 286728336136 + +Cross-validation iteration 2 + +> F8 F7 F6 F5 F4 F3 F2 F1 +> 258582426742713137771155028621733283623416135365078471630162615136662341227482761785123558331153016772554582814217745364 + +Full data set + +Independent feature selection for each iteration Figure 3: See Do be careful where and how you do feature selection. [top] Data leakage due to carrying out feature selection before splitting off the test data (outlined in red), causing the test set to become an implicit part of model training. [centre] How it should be done. [bottom] When using cross-validation, it’s important to carry out feature selection independently for each iteration, based only on the subset of data (shown in blue) used for training during that iteration. 12 Tanks Not tanks Figure 4: See Do avoid learning spurious correlations . The problem of spurious correlations in images, as illustrated by the tank problem. The images on the left are tanks, and those on the right are not tanks. However, the consistent background (blue for tanks, grey for others) means that these images can be classified by merely looking at the colours of pixels towards the top of the images, rather than having to recognise the objects in the images, resulting in a poor model. of these hyperparameters significantly effect the performance of the model, and there is generally no one-size-fits-all. That is, they need to be fitted to your particular data set in order to get the most out of the model. Whilst it may be tempting to fiddle around with hyperparameters until you find something that works, this is not likely to be an optimal approach. It’s much better to use some kind of hyperparameter optimisation strategy, and this is much easier to justify when you write it up. Basic strategies include random search and grid search, but these don’t scale well to large numbers of hyperparameters or to models that are expensive to train, so it’s worth using tools that search for optimal configurations in a more intelligent manner. See Bischl et al. [2023] for further guidance. It is also possible to use AutoML techniques to optimise both the choice of model and its hyperparameters, in addition to other parts of the machine learning pipeline — see Barbudo et al. [2023] for a review. + +3.8 Do avoid learning spurious correlations + +Spurious correlations are features within data which are correlated with the target vari-able, but which have no semantic meaning. They are basically red herrings, and it’s not uncommon for ML models to pick up on them in training, and consequently fail to generalise well. A classic example is the tank problem. Legend 2 has it that the US military were looking to train an ML model that could recognise tanks. However, be-cause the tank pictures used in training were taken during different weather conditions to the non-tank pictures, the model ended up discriminating based on features such as the number of blue pixels in the sky, rather than the presence of a tank (see Figure 4 for + +> 2There is some debate about whether this actually happened: see https://www.gwern.net/Tanks . + +13 an illustration). An ML model that uses such spurious correlations to perform classifica-tion would appear to be very good, in terms of its metric scores, but would not work in practice. More complex data tends to contain more of these spurious correlations, and more complex models have more capacity to overfit spurious correlations. This means that spurious correlations are a particular issue for deep learning, where approaches such as regularisation (see Do keep up with progress in deep learning (and its pitfalls)) and data augmentation (see Do make sure you have enough data) can help mitigate against this. However, spurious correlations can occur in all data sets and models, so it is al-ways worth looking at your trained model to see whether it’s responding to appropriate features within your data — see Do look at your models. + +# 4 How to robustly evaluate models + +In order to contribute to progress in your field, you need to have valid results that you can draw reliable conclusions from. Unfortunately it’s really easy to evaluate ML models unfairly, and, by doing so, muddy the waters of academic progress. So, think carefully about how you are going to use data in your experiments, how you are going to measure the true performance of your models, and how you are going to report this performance in a meaningful and informative way. + +4.1 Do use an appropriate test set + +First of all, always use a test set to measure the generality of an ML model. How well a model performs on the training set is almost meaningless, and a sufficiently complex model can entirely learn a training set yet capture no generalisable knowledge. It’s also important to make sure the data in the test set is appropriate. That is, it should not overlap with the training set and it should be representative of the wider population. For example, consider a photographic data set of objects where the images in the training and test set were collected outdoors on a sunny day. The presence of the same weather conditions means that the test set will not be independent, and by not capturing a broader variety of weather conditions, it will also not be representative. Similar situ-ations can occur when a single piece of equipment is used to collect both the training and test data; if the model overlearns characteristics of the equipment, it will likely not generalise to other pieces of equipment, and this will not be detectable by evaluating it on the test set. If using public datasets to test a model, be wary of Frankenstein datasets , which are assembled from other public datasets and risk overlap with training data. Also be careful when handling datasets that contain multiple data points for each subject; if using these, it’s important to make sure that each subject’s data points are kept together when splitting off the test set or when doing cross-validation. See Roberts et al. [2021] for a revealing account of how a number of these pitfalls led to the failure of the vast majority of Covid-19 detection models to generalise beyond their test sets. 14 4.2 Don’t do data augmentation before splitting your data + +Data augmentation (see Do make sure you have enough data) can be a useful technique for balancing datasets and boosting the generality and robustness of ML models. How-ever, it’s important to do data augmentation only on the training set, and not on data that’s going to be used for testing. Including augmented data in the test set can lead to a number of problems. One problem is that the model may overfit the characteristics of the augmented data, rather than the original samples, and you won’t be able to detect this if your test set also contains augmented data. A more critical problem occurs when data augmentation is applied to the entire data set before it is split into training and test sets. In this scenario, augmented versions of training samples may end up in the test set, which in the worst case can lead to a particularly nefarious form of data leakage in which the test samples are mostly variants of the training samples. For an interesting study of how this problem affected an entire field of research, see Vandewiele et al. [2021]. + +4.3 Do avoid sequential overfitting + +Oddly, one of the most pernicious forms of data leakage doesn’t have a commonly-agreed name 3, so I’m going to refer to it as sequential overfitting. This occurs when you train multiple models in succession, using knowledge gained about each model’s performance to guide the configuration of the next one, and you use the same test set to evaluate each model. Often this is done as an informal process, trying out different models and different hyperparameters until you get good performance on the test set. As such, it is rarely documented, which is one reason why it’s so pernicious. Specifically, the problem lies in using the test set throughout this process, since using the test set to choose between models means that information about the test set implicitly leaks into the training process. See Figure 5 for an illustration of this idea. The consequence is that models gradually overfit the test set; the more times you use the test set, the more the overfitting that occurs. The solution is to either use a validation set (i.e. a separate set of samples that are not directly used in training, but which are used to guide training) or use a holdout data set to test the final model. See Cawley and Talbot [2010] and Hosseini et al. [2020] for more on this. + +4.4 Do evaluate a model multiple times + +Many ML models are stochastic or unstable. That is, if you train them multiple times, or if you make small changes to the training data, then their performance varies signif-icantly. The same is true of using LLMs at inference time. This means that a single evaluation of a model can be unreliable, and may either underestimate or overestimate the model’s true potential. For this reason, it is common to carry out multiple evalua-tions. At training time, there are numerous ways of doing this. For stochastic models, the simplest is to train the same model multiple times using different random seeds and then look at the average performance. A more robust approach is to also vary the + +> 3Though Hosseini et al. [2020] suggested “over-hyping”, from over fitting of hyp er-parameters. + +15 Training set Training Predictive model 1 + +> Test set +> Test Performance + +Train… + +> Training set Training Predictive model 2 +> Test set +> Test Performance + +Tweak… + +> Training set Training Predictive model N +> Test set +> Test Performance + +Tweak… + +# … + +> Training set Training Predictive model N +> Test set Training set Training Predictive model 1 +> Validation set +> Validation Performance + +Train… + +> Training set Training Predictive model 2 +> Validation set +> Validation Performance + +Tweak… + +> Training set Training Predictive model N +> Test set +> Test Performance + +Tweak… + +# … + +> Training set Training Predictive model N +> Validation set Test set + +Figure 5: See Do avoid sequential overfitting. [top] Using the test set repeatedly during model selection results in the test set becoming an implicit part of the training process. [bottom] A validation set should be used instead during model selection, and the test set should only be used once to measure the generality of the final model. data for each model trained. Cross-validation (CV) is a particularly popular way of doing this, and comes in numerous flavours [Arlot et al., 2010], most of which involve splitting the data into a number of folds. When doing CV, it is important to be aware of any dependencies within the data and take these into account. Failure to do so can result in data leakage. For instance, in medical datasets, it is commonplace to have multiple data points for a single subject; to avoid data leakage, these should be kept together within the same fold. Time series data is particularly problematic for CV; see Don’t ignore temporal dependencies in time series data for a discussion of how to handle this. If you’re carrying out hyperparameter optimisation, then you should use 16 nested cross-validation (also known as double cross-validation), which uses an extra loop inside the main cross-validation loop to avoid overfitting the test folds. If some of your data classes are small, then you may need to do stratification , which ensures each class is adequately represented in each fold. In addition to looking at average perfor-mance across multiple evaluations, it is also standard practice to provide some measure of spread or confidence, such as the standard deviation or the 95% confidence interval. + +4.5 Do save some data to evaluate your final model instance + +I’ve used the term model quite loosely, but there is an important distinction between evaluating the potential of a general model (e.g. how well a neural network can solve your problem), and the performance of a particular model instance (e.g. a specific neural network produced by one run of back-propagation). Cross-validation is good at the former, but it’s less useful for the latter. Say, for instance, that you carried out ten-fold cross-validation. This would result in ten model instances. Say you then select the instance with the highest test fold score as the model which you will use in practice. How do you report its performance? Well, you might think that its test fold score is a reliable measure of its performance, but it probably isn’t. First, the amount of data in a single fold is relatively small. Second, the instance with the highest score could well be the one with the easiest test fold, so the evaluation data it contains may not be representative. Consequently, the only way of getting a reliable estimate of the model instance’s generality may be to use another test set. This is also true in situations where the independence of the existing test set may have been compromised, e.g. by using it more than once (see Do avoid sequential overfitting). So, if you have enough data, it’s better to keep some aside and only use it once to provide an unbiased estimate of the final selected model instance. However, it’s worth noting one other option when using cross-validation, and that is to ensemble the model instances (see Do combine models (carefully)). The resulting ensemble will have performance in line with the average as measured through cross-validation, so another test set is not required to measure its performance. On the downside, it will likely have poorer inference time, efficiency and interpretability than a single model instance, so this approach is generally only worth considering if you have very little data. + +4.6 Do choose metrics carefully + +Be careful which metrics you use to evaluate your ML models. For instance, in the case of classification models, the most commonly used metric is accuracy, which is the proportion of samples in the data set that were correctly classified by the model. This works fine if your classes are balanced, i.e. if each class is represented by a similar number of samples within the data set. But many data sets are not balanced, and in this case accuracy can be a very misleading metric. Consider, for example, a data set in which 90% of the samples represent one class, and 10% of the samples represent another class. A binary classifier which always outputs the first class, regardless of its input, would have an accuracy of 90%, despite being completely useless (see Figure 6). In this kind 17 Number of correct classifications Total number of classifications + +> Class F5 F4 F3 F2 F1 +> A58242 +> A27131 +> A71155 +> A62173 +> A36234 +> B35365 +> B47163 +> B26151 +> B62341 +> B48276 +> Correct? Predicted +> TRUE ATRUE ATRUE ATRUE ATRUE AFALSE AFALSE AFALSE AFALSE AFALSE A + +## Always predict class A + +Accuracy = 510 50% Class F5 F4 F3 F2 F1 + +> A58242 +> A27131 +> A71155 +> A62173 +> A36234 +> A35365 +> A47163 +> A26151 +> A62341 +> B48276 +> Correct? Predicted +> TRUE ATRUE ATRUE ATRUE ATRUE ATRUE ATRUE ATRUE ATRUE AFALSE A + +## Always predict class A + +Accuracy = 910 = 90% + +> Number of correct classifications Total number of classifications + +Figure 6: See Do choose metrics carefully . The problem with using accuracy as a performance metric on imbalanced data. Here, a dummy model which always predicts the same class label has an accuracy of 50% or 90% depending on the distribution of class labels within the data. of situation, it would be preferable to use a metric such as F1 score, Cohen’s kappa coefficient ( κ) or Matthews Correlation Coefficient (MCC), all of which are relatively insensitive to class size imbalance. For a broader review of methods for dealing with imbalanced data, see Haixiang et al. [2017]. There are also various pitfalls associated with regression metrics, particularly within the context of time series forecasting; see Hewamalage et al. [2023] for a discussion of these. A well-known example is relying only on the RMSE, which (a bit like accuracy) is susceptible to assigning high value to models which always predict no change. Also see Do report performance in multiple ways. + +4.7 Do consider model fairness + +Overall performance metrics are not the only important measures of how good a model is. If a model is to be deployed within the real world, another important measure is fairness. There are various definitions of fairness, but in a nutshell, it’s about making sure that the model doesn’t treat its human subjects unequally with regard to characteristics such as gender, ethnicity, income or personal politics. This is also referred to as algorithmic bias , and there are many examples of models being biased towards or against particular groups of people. A common source of unfairness is using an unrepresentative dataset to train an ML model. For instance, if a medical diagnosis model is trained on data from a single country, then the data may be biased towards the majority ethnicity, 18 and the model may not operate fairly when exposed to users from other ethnicities. However, unfairness can also come from other sources, including subconscious bias during data preparation and the inductive biases of the model. Regardless of the source, it is important to understand any resulting biases, and ideally take steps to mitigate against them (e.g. applying data augmentation to minority samples — see Do make sure you have enough data). There are many different fairness metrics, so part of the puzzle is working out which are most relevant to your modelling context; see Caton and Haas [2024] for a review. + +4.8 Don’t ignore temporal dependencies in time series data + +Time series data is unlike many other kinds of data in that the order of the data points is important. Many of the pitfalls in handling time series data are a result of ignoring this fact. Most notably, time series data are subject to a particular kind of data leakage (see Don’t allow test data to leak into the training process) known as look ahead bias .This occurs when some or all of the data points used to train the model occur later in the time series than those used to test the model. In effect, this can allow knowledge of the future to leak into training, and this can then bias the test performance. A situation where this commonly occurs is when standard cross-validation (see Do evaluate a model multiple times) is applied to time series data, since it results in the training folds in all but one of the cross-validation iterations containing data that is in the future relative to the test fold. This can be avoided by using special forms of cross-validation that respect temporal dependencies, such as blocked cross-validation , though whether this is necessary depends to some extent on the nature of the time series data, e.g. whether it is stationary or non-stationary. See Cerqueira et al. [2020] and Wang and Ruf [2022] for more on this. Look ahead bias can also result from carrying out data-dependent preprocessing operations before splitting off the test data; see Figure 7 for a simple example of this, but also see Do be careful where and how you do feature selection. + +# 5 How to compare models fairly + +Comparing models is the basis of academic research, but it’s surprisingly difficult to get it right. If you carry out a comparison unfairly, and publish it, then other researchers may subsequently be led astray. So, do make sure that you evaluate different models within the same context, do explore multiple perspectives, and do use make correct use of statistical tests. + +5.1 Don’t assume a bigger number means a better model + +It’s not uncommon for a paper to state something like “In previous research, accuracies of up to 94% were reported. Our model achieved 95%, and is therefore better.” There are various reasons why a higher figure does not imply a better model. For instance, if the models were trained or evaluated on different partitions of the same data set, then small 19 time 5000 4000 3000 2000 1000 0-1000 -2000 time 100.75 0.5 0.25 + +> Train Test Train Test +> time 5000 4000 3000 2000 1000 0-1000 -2000 time +> Train Test +> 010.75 0.5 0.25 1.25 + +Figure 7: See Don’t ignore temporal dependencies in time series data. [top] A time series is scaled to the interval [0 , 1] before splitting off the test data (shown in red). This could allow the model to infer that values will increase in the future, causing a potential look ahead bias. [bottom] Instead, the data should be split before doing scaling, so that information about the range of the test data can’t leak into the training data. differences in performance may be due to this. If the datasets had different degrees of class imbalance, then the difference in accuracy could merely reflect this (see Do choose metrics carefully). If they used different data sets entirely, then this may account for even large differences in performance. Another reason for unfair comparisons is the failure to carry out the same amount of hyperparameter optimisation (see Do optimise your model’s hyperparameters) when comparing models; for instance, if one model has default settings and the other has been optimised, then the comparison won’t be fair. For these reasons, and others, comparisons based on published figures should always 20 be treated with caution. To be sure of a fair comparison between two approaches, you should freshly implement all the models you’re comparing, optimise each one to the same degree, carry out multiple evaluations (see Do evaluate a model multiple times), and then use statistical tests (see Do use statistical tests when comparing models) to determine whether the differences in performance are significant. A further complication when comparing foundation models (see Do keep up with progress in deep learning (and its pitfalls)) it that the original training data is often unknown; consequently it may be impossible to ensure that the test set is independent of the training data, and therefore a fair basis for comparison. + +5.2 Do use meaningful baselines + +When introducing a new modelling approach, it is essential to compare against estab-lished approaches. These are commonly referred to as baseline models , or just base-lines. It is important that these baselines are selected so that they provide a meaningful basis for comparison. Baselines are often simpler than the new approach, and are cho-sen to demonstrate that any complexity in the new model is necessary. For example, if you’re extending model X, then it makes sense to use model X as a baseline. However, it also makes sense to use other simpler models. For instance, if you’re developing a deep learning approach that uses tabular data, then you should also compare against simpler models like decision trees and SVMs to show that a more complex approach is justified. If you’re solving a regression problem, then you should also consider using simple base-lines like logistic regression. The simplest baselines are known as na¨ ıve baselines and are used to show that your model is not doing something trivial. An illustrative example of why these are necessary is described in Hewamalage et al. [2023], where a complex transformer model designed for time series forecasting is shown to perform worse than a na¨ ıve baseline that always forecasts the next value in a time series to be the same as the previous value. This kind of na¨ ıve baseline, in which there is no real decision making process, is also known as a dummy model . Another example is a classifier that al-ways outputs the most frequent class label (as described in Do choose metrics carefully). In addition to simple baselines, it’s also important to compare against state-of-the-art (SOTA ) models. Otherwise you may be asked something like “Why are you extending model X when model Y is known to be better than model X?” + +5.3 Do use statistical tests when comparing models + +If you want to convince people that your model is better than someone else’s, then a statistical test can be a useful tool. Broadly speaking, there are two categories of tests for comparing individual ML models. The first is used to compare individual model instances, e.g. two trained decision trees. For example, McNemar’s test is a fairly common choice for comparing two classifiers, and works by comparing the classifiers’ output labels for each sample in the test set (so do remember to record these). The second category of tests are used to compare two models more generally, e.g. whether a decision tree or a neural network is a better fit for the data. These require multiple evaluations 21 of each model, which you can get by using cross-validation or repeated resampling (or, if your training algorithm is stochastic, multiple repeats using the same data). The test then compares the two resulting distributions. Student’s T test is a common choice for this kind of comparison, but it’s only reliable when the distributions are normally distributed, which is often not the case. A safer bet is Mann-Whitney’s U test, since this does not assume that the distributions are normal. For more information, see Raschka [2020] and Carrasco et al. [2020]. Also see Do correct for multiple comparisons and Do be careful when reporting statistical significance. + +5.4 Do correct for multiple comparisons + +Things get a bit more complicated when you want to use statistical tests to compare more than two models, since doing multiple pairwise tests is a bit like using the test set multiple times — it can lead to overly-optimistic interpretations of significance. Basically, each time you carry out a comparison between two models using a statistical test, there’s a probability that it will discover significant differences where there aren’t any. This is represented by the confidence level of the test, usually set at 95%: meaning that 1 in 20 times it will give you a false positive. For a single comparison, this may be a level of uncertainty you can live with. However, it accumulates. That is, if you do 20 pairwise tests with a confidence level of 95%, one of them is likely to give you the wrong answer. This is known as the multiplicity effect , and is an example of a broader issue in data science known (at least when done intentionally) as data dredging or p-hacking — see Stefan and Sch¨ onbrodt [2023]. To address this problem, you can apply a correction for multiple tests. The most common approach is the Bonferroni correction, a very simple method that lowers the significance threshold based on the number of tests that are being carried out; see Salzberg [1997] for a gentle introduction. However, there are numerous other approaches, and there is also some debate about when and where these corrections should be applied; for an accessible overview, see Streiner [2015]. + +5.5 Don’t always believe results from community benchmarks + +In certain problem domains, it has become commonplace to use benchmark data sets to evaluate new ML models. The idea is that, because everyone is using the same data to train and test their models, then comparisons will be more transparent. Unfortunately this approach has some major drawbacks. First, if access to the test set is unrestricted, then you can’t assume that people haven’t used it as part of the training process. This is known as “training to the test set”, and leads to results that are heavily over-optimistic. A more subtle problem is that, even if everyone who uses the data only uses the test set once, collectively the test set is being used many times by the community. In effect, by comparing lots of models on the same test set, it becomes increasingly likely that the best model just happens to over-fit the test set, and doesn’t necessarily generalise any better than the other models (see Do correct for multiple comparisons and Do avoid sequential overfitting). For these, and other reasons, you should be careful how much you read into results from a benchmark data set, and don’t assume that a small increase 22 in performance is significant. This is particularly the case where foundation models (see Do keep up with progress in deep learning (and its pitfalls)) are used, since it is possible that their training data included the test sets from community benchmarks. See Paullada et al. [2021] for a wider discussion of issues surrounding the use of shared datasets. Also see Do report performance in multiple ways. + +5.6 Do combine models (carefully) + +Whilst this section focuses on comparing models, it’s good to be aware that ML is not always about choosing between models. Often it makes sense to use combinations of models. Different ML models explore different trade-offs; by combining them, you can sometimes compensate for the weaknesses of one model by using the strengths of another model, and vice versa. Ensembles are a well-established group of composite models. There are lots of ensemble learning approaches — see Dong et al. [2020] for a review — but they can be roughly divided into those that form ensembles out of the same base model type (examples include random forests, bagging and boosting) and those that combine different types of ML model. An example of the latter is stacked generalisation (or stacking), where a model is trained to aggregate the outputs of a group of base models. However, ensembles are not the only kind of composition. Another, increasingly common, form of composition occurs when embedding models + +(such as autoencoders or foundation models such as BERT) are used to provide input to other models. When using stacking or embedding, it’s important to ensure that no data leaks (see Don’t allow test data to leak into the training process) occur, i.e. that the test data used to measure the performance of the composite model is not used in the training of any of its components. This is a common pitfall, especially when the model components are trained on overlapping data. To reduce the likelihood of sequential overfitting (see Do avoid sequential overfitting), it is also advisable to use a separate test set to evaluate the composite model. + +# 6 How to report your results + +The aim of academic research is not self-aggrandisement, but rather an opportunity to contribute to knowledge. In order to effectively contribute to knowledge, you need to provide a complete picture of your work, covering both what worked and what didn’t. ML is often about trade-offs — it’s very rare that one model is better than another in every way that matters — and you should try to reflect this with a nuanced and considered approach to reporting results and conclusions. + +6.1 Do be transparent + +First of all, always try to be transparent about what you’ve done, and what you’ve discovered, since this will make it easier for other people to build upon your work. In particular, it’s good practice to share your models in an accessible way. For instance, if you used a script to implement all your experiments, then share the script when you 23 publish the results. This means that other people can easily repeat your experiments, which adds confidence to your work. It also makes it a lot easier for people to compare models, since they no longer have to reimplement everything from scratch in order to ensure a fair comparison. Knowing that you will be sharing your work also encourages you to be more careful, document your experiments well, and write clean code, which benefits you as much as anyone else. It’s also worth noting that issues surrounding reproducibility are gaining prominence in the ML community, so in the future you may not be able to publish work unless your workflow is adequately documented and shared — for example, see Pineau et al. [2021]. Checklists (Do use a machine learning checklist) are useful for knowing what to include in your workflow. You might also find experiment tracking frameworks, such as MLflow [Chen et al., 2020], useful for recording your workflow. + +6.2 Do report performance in multiple ways + +One way to achieve better rigour when evaluating and comparing models is to use mul-tiple data sets. This helps to overcome any deficiencies associated with individual data sets (see Don’t always believe results from community benchmarks) and allows you to present a more complete picture of your model’s performance. It’s also good practice to report multiple metrics for each data set, since different metrics can present different perspectives on the results, and increase the transparency of your work. For example, if you use accuracy, it’s also a good idea to include metrics that are less sensitive to class imbalances (see Do choose metrics carefully). In domains such as medicine and security, it’s important to know where errors are being made; for example, when your model gets things wrong, is it more inclined to false positives or false negatives? Metrics that summarise everything in one number, such as accuracy, give no insight into this. So, it’s important to also include partial metrics such as precision and recall, or sensi-tivity and specificity, since these do provide insight into the types of errors your model produces. And make sure it’s clear which metrics you are using. For instance, if you report F-scores, be clear whether this is F1, or some other balance between precision and recall. If you report AUC, indicate whether this is the area under the ROC curve or the PR curve. For a broader discussion, see Blagec et al. [2020]. + +6.3 Don’t generalise beyond the data + +It’s important not to present invalid conclusions, since this can lead other researchers astray. A common mistake is to make general statements that are not supported by the data used to train and evaluate models. For instance, if your model does really well on one data set, this does not mean that it will do well on other data sets. Whilst you can get more robust insights by using multiple data sets (see Do report performance in multiple ways), there will always be a limit to what you can infer from any experimental study. There are numerous reasons for this (see Paullada et al. [2021]), many of which are to do with how datasets are curated. One common issue is bias, or sampling error :that the data is not sufficiently representative of the real world. Another is overlap: 24 multiple data sets may not be independent, and may have similar biases. There’s also the issue of quality: and this is a particular issue in deep learning datasets, where the need for quantity of data limits the amount of quality checking that can be done. So, in short, don’t overplay your findings, and be aware of their limitations. + +6.4 Do be careful when reporting statistical significance + +I’ve already discussed statistical tests (see Do use statistical tests when comparing mod-els), and how they can be used to determine differences between ML models. However, statistical tests are not perfect. Some are conservative, and tend to under-estimate sig-nificance; others are liberal, and tend to over-estimate significance. This means that a positive test doesn’t always indicate that something is significant, and a negative test doesn’t necessarily mean that something isn’t significant. Then there’s the issue of us-ing a threshold to determine significance; for instance, a 95% confidence threshold (i.e. when the p-value < 0.05) means that 1 in 20 times a difference flagged as significant won’t be significant. In fact, statisticians are increasingly arguing that it is better not to use thresholds, and instead just report p-values and leave it to the reader to inter-pret these [Betensky, 2019]. Beyond statistical significance, another thing to consider is whether the difference between two models is actually important. If you have enough samples, you can always find significant differences, even when the actual difference in performance is miniscule. To give a better indication of whether something is im-portant, you can measure effect size . There are a range of approaches used for this: Cohen’s d statistic is probably the most common, but more robust approaches, such as Kolmogorov-Smirnov, are preferable. For more on effect size and reporting statistical significance, see Aguinis et al. [2021]. You might also consider using Bayesian statistics; although there’s less guidance and tools support available, these theoretically have a lot going for them, and they avoid many of the pitfalls associated with traditional statistical tests — see Benavoli et al. [2017] for more info. + +6.5 Do look at your models + +Trained models contain a lot of useful information. Unfortunately many authors just report the performance metrics of a trained model, without giving any insight into what it actually learnt. Remember that the aim of research is not to get a slightly higher accuracy than everyone else. Rather, it’s to generate knowledge and understanding and share this with the research community. If you can do this, then you’re much more likely to get a decent publication out of your work. So, do look inside your models and do try to understand how they reach a decision. For relatively simple models like decision trees, it can also be beneficial to provide visualisations of your models, and most libraries have functions that will do this for you. For more complex models, there are a range of explainable AI (XAI) techniques that can be used. Some of these are model-specific, and others are model-agnostic. Well-established examples of the latter are LIME and SHAP; both give insights into which features are important for a model. For CNNs and vision transformers, a common approach is to use saliency maps , which show 25 Figure 8: See Do look at your models. Using saliency maps to analyse vision-based deep learning models. Imagine these two maps (in red) were generated for the image shown in the centre, for two different deep learning models trained on the kind of tank recognition data mentioned in Do avoid learning spurious correlations. Darker colours indicate features that are of greater importance to the model, so the model on the left (which predominantly focuses on the components of the tank) is likely to generalise much better than the one on the right (which predominantly focuses on the background of the image). the importance of different parts of an input image — see Figure 8 for an illustrative example. Grad-CAM is a popular technique for generating these, but there are plenty of other methods too. For non-vision transformers, a common approach is to visualise attention weights. See Dwivedi et al. [2023] for a survey of XAI techniques, and Ali et al. [2023] for a discussion of the limitations of current approaches. Whilst XAI techniques can give you useful insights into a model’s behaviour, it’s important to bear in mind that they are unlikely to tell you exactly what a model is doing. This is particularly the case for deep learning models (see Don’t assume deep learning will be the best approach), whose complexity makes their behaviour inherently difficult to analyse. For complex models, ablation studies Meyes et al. [2019] can also be useful. This involves successively removing parts of the model to see what is important, and can result in a simpler model which is more amenable to analysis. + +6.6 Do use a machine learning checklist + +This guide aims to give an appreciation of the main things that can go wrong dur-ing machine learning, plus some guidance on how to avoid these things going wrong. Checklists, on the other hand, are designed to take you more formally through the ML pipeline and encourage you to document (and more importantly, think about) how your implementation decisions support a meaningful outcome. In some domains, e.g. certain fields of medicine, it is compulsory to complete a checklist before submitting a paper for publication. However, beyond their quality assurance role, checklists are arguably most useful at the start of a study when it comes to planning an ML pipeline. Since I’m one of the authors, I’d particularly encourage you to look at REFORMS [Kapoor et al., 2024], which is a combined checklist and set of consensus-based recommendations for doing ML-based science (although much of it is also applicable to ML practice more generally). Other, more domain-specific, checklists are also available. 26 7 Final thoughts + +ML is becoming an important part of people’s lives, yet the practice of ML is arguably in its infancy. There are many easy-to-make mistakes that can cause an ML model to appear to perform well, when in reality it does not. In turn, this has the potential to misinform when these models are published, and the potential to cause harm if these models are ever deployed. This guide describes the most common of these mistakes, and also touches upon more general issues of good practice in ML, such as fairness, transparency and the avoidance of bias. It also offers advice on avoiding these pitfalls. However, new threats continue to emerge as new approaches to ML are developed, and it is therefore important for users of ML to remain vigilant. This is the nature of a fast-moving research area — the theory of how to do ML almost always lags behind the practice, practitioners will always disagree about the best ways of doing things, and what we think is correct today may not be correct tomorrow. You can find more on ML pitfalls at my Substack, Fetch Decode Execute. + +# Acknowledgements + +Many thanks to everyone who gave me feedback on the draft manuscript, to everyone who has since sent me suggestions for revisions and new content, and to the editor and peer reviewers of the version published in Patterns . + +# References + +Where available, preprint URLs are also included for papers that are not open access. H. Aguinis, M. Vassar, and C. Wayant. On reporting and interpreting statistical signif-icance and p values in medical research. BMJ Evidence-Based Medicine , 26(2):39–42, 2021. http://doi.org/10.1136/bmjebm-2019-111264. S. Ali, T. Abuhmed, S. El-Sappagh, K. Muhammad, J. M. Alonso-Moral, R. Con-falonieri, R. Guidotti, J. Del Ser, N. D´ ıaz-Rodr´ ıguez, and F. Herrera. Ex-plainable artificial intelligence (xai): What we know and what is left to at-tain trustworthy artificial intelligence. Information fusion , 99:101805, 2023. https://doi.org/10.1016/j.inffus.2023.101805. S. Arlot, A. Celisse, et al. A survey of cross-validation procedures for model selection. + +Statistics surveys , 4:40–79, 2010. https://doi.org/10.1214/09-SS054. D. Arp, E. Quiring, F. Pendlebury, A. Warnecke, F. Pierazzi, C. Wressnegger, L. Cav-allaro, and K. Rieck. Dos and don’ts of machine learning in computer security. In + +31st USENIX Security Symposium (USENIX Security 22) , pages 3971–3988, 2022. https://www.usenix.org/system/files/sec22-arp.pdf. 27 R. Barbudo, S. Ventura, and J. R. Romero. Eight years of automl: categorisation, review and trends. Knowledge and Information Systems , 65(12):5097–5149, 2023. https://doi.org/10.1007/s10115-023-01935-1. A. Benavoli, G. Corani, J. Demˇ sar, and M. Zaffalon. Time for a change: a tutorial for comparing multiple classifiers through bayesian analysis. The Journal of Machine Learning Research , 18(1):2653–2688, 2017. https://jmlr.org/papers/v18/16-305.html. R. A. Betensky. The p-value requires context, not a threshold. The American Statisti-cian , 73(sup1):115–117, 2019. https://doi.org/10.1080/00031305.2018.1529624. B. Bischl, M. Binder, M. Lang, T. Pielok, J. Richter, S. Coors, J. Thomas, T. Ullmann, M. Becker, A.-L. Boulesteix, et al. Hyperparameter optimiza-tion: Foundations, algorithms, best practices, and open challenges. Wiley Inter-disciplinary Reviews: Data Mining and Knowledge Discovery , 13(2):e1484, 2023. https://doi.org/10.1002/widm.1484. K. Blagec, G. Dorffner, M. Moradi, and M. Samwald. A critical analysis of met-rics used for measuring progress in artificial intelligence. Preprint at arXiv , 2020. https://arxiv.org/abs/2008.02577. A. D. Blaom, F. Kiraly, T. Lienart, Y. Simillides, D. Arenas, and S. J. Vollmer. Mlj: A julia package for composable machine learning. Journal of Open Source Software , 5 (55):2704, 2020. https://doi.org/10.21105/joss.02704. J. Cai, J. Luo, S. Wang, and S. Yang. Feature selection in ma-chine learning: A new perspective. Neurocomputing , 300:70–79, 2018. https://doi.org/10.1016/j.neucom.2017.11.077. J. Carrasco, S. Garc´ ıa, M. Rueda, S. Das, and F. Herrera. Recent trends in the use of statistical tests for comparing swarm and evolutionary computing algorithms: Practi-cal guidelines and a critical review. Swarm and Evolutionary Computation , 54:100665, 2020. https://doi.org/10.1016/j.swevo.2020.100665. S. Caton and C. Haas. Fairness in machine learning: A survey. ACM Computing Surveys ,56(7):1–38, 2024. https://doi.org/10.1145/3616865. G. C. Cawley and N. L. Talbot. On over-fitting in model selection and subsequent selec-tion bias in performance evaluation. The Journal of Machine Learning Research , 11: 2079–2107, 2010. https://www.jmlr.org/papers/volume11/cawley10a/cawley10a.pdf. V. Cerqueira, L. Torgo, and I. Mozetiˇ c. Evaluating time series forecasting models: An empirical study on performance estimation methods. Machine Learning , 109(11): 1997–2028, 2020. https://doi.org/10.1007/s10994-020-05910-7. A. Chen, A. Chow, A. Davidson, A. DCunha, A. Ghodsi, S. A. Hong, A. Konwinski, C. Mewald, S. Murching, T. Nykodym, et al. Developments in mlflow: A system to accelerate the machine learning lifecycle. In Proceedings of the fourth international + +28 workshop on data management for end-to-end machine learning , pages 1–4, 2020. https://doi.org/10.1145/3399579.3399867. P.-O. Cˆ ot´ e, A. Nikanjam, N. Ahmed, D. Humeniuk, and F. Khomh. Data clean-ing and machine learning: a systematic literature review. Automated Software Engineering , 31(2):54, 2024. https://doi.org/10.1007/s10515-024-00453-w (preprint: https://doi.org/10.48550/arXiv.2310.01765). V. Cox. Exploratory data analysis. In Translating Statistics to Make Decisions , pages 47–74. Springer, 2017. X. Dong, Z. Yu, W. Cao, Y. Shi, and Q. Ma. A survey on ensemble learning. Frontiers of Computer Science , 14(2):241–258, 2020. https://doi.org/10.1007/s11704-019-8208-z. R. Dwivedi, D. Dave, H. Naik, S. Singhal, R. Omer, P. Patel, B. Qian, Z. Wen, T. Shah, G. Morgan, et al. Explainable ai (xai): Core ideas, techniques, and solutions. ACM Computing Surveys , 55(9):1–33, 2023. https://doi.org/10.1145/3561048. T. Emmanuel, T. Maupong, D. Mpoeleng, T. Semong, B. Mphago, and O. Tabona. A survey on missing data in machine learning. Journal of Big data , 8:1–37, 2021. https://doi.org/10.1186/s40537-021-00516-9. E. Gibney. Is AI fuelling a reproducibility crisis in science? Nature , 608:250–251, 2022. https://doi.org/10.1038/d41586-022-02035-w. L. Grinsztajn, E. Oyallon, and G. Varoquaux. Why do tree-based models still outperform deep learning on typical tabular data? Advances in neural information processing systems , 35:507–520, 2022. https://openreview.net/pdf?id=Fp7 phQszn. G. Haixiang, L. Yijing, J. Shang, G. Mingyun, H. Yuanyue, and G. Bing. Learning from class-imbalanced data: Review of methods and applications. Expert Systems with Applications , 73:220–239, 2017. https://doi.org/10.1016/j.eswa.2016.12.035. X. Han, Z. Zhang, N. Ding, Y. Gu, X. Liu, Y. Huo, J. Qiu, Y. Yao, A. Zhang, L. Zhang, et al. Pre-trained models: Past, present and future. AI Open , 2:225–250, 2021. https://doi.org/10.1016/j.aiopen.2021.08.002. H. Hewamalage, K. Ackermann, and C. Bergmeir. Forecast evaluation for data scientists: common pitfalls and best practices. Data Mining and Knowledge Discovery , 37(2): 788–832, 2023. https://doi.org/10.1007/s10618-022-00894-5. M. Hosseini, M. Powell, J. Collins, C. Callahan-Flintoft, W. Jones, H. Bow-man, and B. Wyble. I tried a bunch of things: The dangers of unexpected overfitting in classification of brain data. Neuroscience & Biobehavioral Re-views , 119:456–467, 2020. https://doi.org/10.1016/j.neubiorev.2020.09.036 (preprint: https://www.biorxiv.org/content/10.1101/078816v2.abstract). 29 G. Iglesias, E. Talavera, ´A. Gonz´ alez-Prieto, A. Mozo, and S. G´ omez-Canaval. Data augmentation techniques in time series domain: a survey and taxonomy. Neural Com-puting and Applications , 35(14):10123–10145, 2023. https://doi.org/10.1007/s00521-023-08459-3. S. Kapoor and A. Narayanan. Leakage and the reproducibility crisis in machine-learning-based science. Patterns , 4(9):100804, 2023. ISSN 2666-3899. https://doi.org/10.1016/j.patter.2023.100804. S. Kapoor, E. M. Cantrell, K. Peng, T. H. Pham, C. A. Bail, O. E. Gundersen, J. M. Hofman, J. Hullman, M. A. Lones, M. M. Malik, P. Nanayakkara, R. A. Poldrack, I. D. Raji, M. Roberts, M. J. Salganik, M. Serra-Garcia, B. M. Stew-art, G. Vandewiele, and A. Narayanan. REFORMS: Consensus-based recommen-dations for machine-learning-based science. Science Advances , 10(18):eadk3452, 2024. https://doi.org/10.1126/sciadv.adk3452. P. Karande, B. Gallagher, and T. Y.-J. Han. A strategic approach to machine learning for material science: How to tackle real-world challenges and avoid pitfalls. Chemistry of Materials , 34(17):7650–7665, 2022. https://doi.org/10.1021/acs.chemmater.2c01333. S. Khan, M. Naseer, M. Hayat, S. W. Zamir, F. S. Khan, and M. Shah. Transformers in vision: A survey. ACM computing surveys (CSUR) , 54(10s):1–41, 2022. https://doi.org/10.1145/3505244 (preprint: https://doi.org/10.48550/arXiv.2101.01169). D. Kreuzberger, N. K¨ uhl, and S. Hirschl. Machine learning operations (MLOps): Overview, definition, and architecture. IEEE access , 2023. https://doi.org/10.1109/ACCESS.2023.3262138. M. Kuhn and H. Wickham. Tidymodels: a collection of packages for modeling and machine learning using tidyverse principles , 2020. https://www.tidymodels.org. H. Li, Y. Chen, J. Luo, Y. Kang, X. Zhang, Q. Hu, C. Chan, and Y. Song. Privacy in large language models: Attacks, defenses and future directions. Preprint at arXiv ,2023. https://doi.org/10.48550/arXiv.2310.10383. Z. Li, F. Liu, W. Yang, S. Peng, and J. Zhou. A survey of convolutional neural net-works: analysis, applications, and prospects. IEEE transactions on neural networks and learning systems , 2021. https://doi.org/10.1109/TNNLS.2021.3084827 (preprint: https://doi.org/10.48550/arXiv.2004.02806). T. Liao, R. Taori, I. D. Raji, and L. Schmidt. Are we learning yet? a meta review of evaluation failures across machine learning. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2) , 2021. https://openreview.net/forum?id=mPducS1MsEK. T. Lin, Y. Wang, X. Liu, and X. Qiu. A survey of transformers. AI Open , 2022. https://doi.org/10.1016/j.aiopen.2022.10.001. 30 M. A. Lones. Avoiding machine learning pitfalls. Patterns , 2024. https://doi.org/10.1016/j.patter.2024.101046. M. M. Malik. A hierarchy of limitations in machine learning. Preprint at arXiv , 2020. https://doi.org/10.48550/arXiv.2002.05193. R. Meyes, M. Lu, C. W. de Puiseau, and T. Meisen. Ablation studies in artificial neural networks. Preprint at arXiv , 2019. https://doi.org/10.48550/arXiv.1901.08644. C. Molnar, G. K¨ onig, J. Herbinger, T. Freiesleben, S. Dandl, C. A. Scholbeck, G. Casalic-chio, M. Grosse-Wentrup, and B. Bischl. General pitfalls of model-agnostic interpreta-tion methods for machine learning models. In International Workshop on Extending Explainable AI Beyond Deep Models and Classifiers , pages 39–68. Springer, 2020. https://doi.org/10.1007/978-3-031-04083-2 4. A. Paullada, I. D. Raji, E. M. Bender, E. Denton, and A. Hanna. Data and its (dis)contents: A survey of dataset development and use in ma-chine learning research. Patterns , 2(11):100336, 2021. ISSN 2666-3899. https://doi.org/10.1016/j.patter.2021.100336. J. Pineau, P. Vincent-Lamarre, K. Sinha, V. Larivi` ere, A. Beygelzimer, F. d’Alch´ e Buc, E. Fox, and H. Larochelle. Improving reproducibility in machine learning research (a report from the neurips 2019 reproducibility program). Journal of machine learn-ing research , 22(164):1–20, 2021. https://www.jmlr.org/papers/volume22/20-303/20-303.pdf. S. Raschka. Model evaluation, model selection, and algorithm selection in machine learning. Preprint at arXiv , 2020. https://arxiv.org/abs/1811.12808. M. Roberts, D. Driggs, M. Thorpe, J. Gilbey, M. Yeung, S. Ursprung, A. I. Aviles-Rivero, C. Etmann, C. McCague, L. Beer, et al. Common pitfalls and recommen-dations for using machine learning to detect and prognosticate for covid-19 using chest radiographs and ct scans. Nature Machine Intelligence , 3(3):199–217, 2021. https://doi.org/10.1038/s42256-021-00307-0. C. Rudin. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelli-gence , 1(5):206–215, 2019. https://doi.org/10.1038/s42256-019-0048-x (preprint: https://doi.org/10.48550/arXiv.1811.10154). S. L. Salzberg. On comparing classifiers: Pitfalls to avoid and a recom-mended approach. Data mining and knowledge discovery , 1(3):317–328, 1997. https://doi.org/10.1023/A:1009752403260. J. Schmidhuber. Deep learning in neural networks: An overview. Neural net-works , 61:85–117, 2015. https://doi.org/10.1016/j.neunet.2014.09.003 (preprint: https://doi.org/10.48550/arXiv.1404.7828). 31 D. Sculley, G. Holt, D. Golovin, E. Davydov, T. Phillips, D. Ebner, V. Chaudhary, M. Young, J.-F. Crespo, and D. Dennison. Hidden technical debt in machine learning systems. Advances in neural information processing systems , 28:2503–2511, 2015. https://papers.nips.cc/paper/2015/file/86df7dcfd896fcaf2674f757a2463eba-Paper.pdf. S. Shankar, R. Garcia, J. M. Hellerstein, and A. G. Parameswaran. Opera-tionalizing machine learning: An interview study. Preprint at arXiv , 2022. https://doi.org/10.48550/arXiv.2209.09125. A. M. Stefan and F. D. Sch¨ onbrodt. Big little lies: A compendium and simu-lation of p-hacking strategies. Royal Society Open Science , 10(2):220346, 2023. https://doi.org/10.1098/rsos.220346. L. M. Stevens, B. J. Mortazavi, R. C. Deo, L. Curtis, and D. P. Kao. Recommendations for reporting machine learning analyses in clinical re-search. Circulation: Cardiovascular Quality and Outcomes , 13(10):e006556, 2020. https://doi.org/10.1161/CIRCOUTCOMES.120.006556. D. L. Streiner. Best (but oft-forgotten) practices: the multiple problems of multiplic-ity—whether and how to correct for many statistical tests. The American journal of clinical nutrition , 102(4):721–728, 2015. https://doi.org/10.3945/ajcn.115.113548. B. Van Giffen, D. Herhausen, and T. Fahse. Overcoming the pitfalls and perils of algo-rithms: A classification of machine learning biases and mitigation methods. Journal of Business Research , 144:93–106, 2022. https://doi.org/10.1016/j.jbusres.2022.01.076. G. Vandewiele, I. Dehaene, G. Kov´ acs, L. Sterckx, O. Janssens, F. Ongenae, F. De Backere, F. De Turck, K. Roelens, J. Decruyenaere, S. Van Hoecke, and T. Demeester. Overly optimistic prediction results on imbalanced data: a case study of flaws and benefits when applying over-sampling. Artificial Intelligence in Medicine , 111:101987, 2021. https://doi.org/10.1016/j.artmed.2020.101987 (preprint: https://arxiv.org/abs/2001.06296). G. Varoquaux, L. Buitinck, G. Louppe, O. Grisel, F. Pedregosa, and A. Mueller. Scikit-learn: Machine learning without learning the machin-ery. GetMobile: Mobile Computing and Communications , 19(1):29–33, 2015. https://doi.org/10.1145/2786984.2786995. W. Wang and J. Ruf. Information leakage in backtesting. Available at SSRN 3836631 ,2022. https://doi.org/10.2139/ssrn.3836631. Z. Wang, P. Wang, K. Liu, P. Wang, Y. Fu, C.-T. Lu, C. C. Aggarwal, J. Pei, and Y. Zhou. A comprehensive survey on data augmentation. Preprint at arXiv , 2024. https://doi.org/10.48550/arXiv.2405.09591. 32 S. Whalen, J. Schreiber, W. S. Noble, and K. S. Pollard. Navigating the pitfalls of applying machine learning in genomics. Nature Reviews Genetics , 23(3):169–181, 2022. https://doi.org/10.1038/s41576-021-00434-9. D. H. Wolpert. The supervised learning no-free-lunch theorems. Soft computing and industry , pages 25–42, 2002. https://doi.org/10.1007/978-1-4471-0123-9 3. A. Zeng, M. Chen, L. Zhang, and Q. Xu. Are transformers effec-tive for time series forecasting? In Proceedings of the AAAI con-ference on artificial intelligence , volume 37, pages 11121–11128, 2023. https://ojs.aaai.org/index.php/AAAI/article/view/26317/26089. A. Zhang, Z. C. Lipton, M. Li, and A. J. Smola. Dive into deep learning . Cambridge University Press, 2023. https://d2l.ai. C. Zhou, Q. Li, C. Li, J. Yu, Y. Liu, G. Wang, K. Zhang, C. Ji, Q. Yan, L. He, et al. A comprehensive survey on pretrained foundation models: A history from bert to chatgpt. Preprint at arXiv , 2023. https://doi.org/10.48550/arXiv.2302.09419. J.-J. Zhu, M. Yang, and Z. J. Ren. Machine learning in environmental research: common pitfalls and best practices. Environmental Science & Technology , 57(46):17671–17689, 2023. https://doi.org/10.1021/acs.est.3c00026. 33 diff --git a/docs/evidence/sculley_2015_hidden_technical_debt.md b/docs/evidence/sculley_2015_hidden_technical_debt.md index b084f17..4b13551 100644 --- a/docs/evidence/sculley_2015_hidden_technical_debt.md +++ b/docs/evidence/sculley_2015_hidden_technical_debt.md @@ -1,18 +1,159 @@ Source: https://papers.nips.cc/paper_files/paper/2015/file/86df7dcfd896fcaf2674f757a2463eba-Paper.pdf -Title: "Hidden Technical Debt in Machine Learning Systems" — D. Sculley, Gary Holt, Daniel Golovin, Eugene Davydov, Todd Phillips, Dietmar Ebner, Vinay Chaudhary, Michael Young, Jean-François Crespo, Dan Dennison (Google, Inc.), NIPS 2015 -Fetched-via: PDF downloaded from papers.nips.cc, pages 1-2 transcribed by hand from the rendered pages -Fetch-status: verbatim excerpts (abstract + Entanglement section); subscripts rendered as plain text (x1, xn+1) +Title: "Hidden Technical Debt in Machine Learning Systems" -- D. Sculley, Gary Holt, Daniel Golovin, Eugene Davydov, Todd Phillips, Dietmar Ebner, Vinay Chaudhary, Michael Young, Jean-Francois Crespo, Dan Dennison (Google, Inc.), NIPS 2015 +Fetched-via: curl https://r.jina.ai/, 2026-08-14 +Fetch-status: verbatim, full paper (9 pages) -# Hidden Technical Debt in Machine Learning Systems (excerpts) +Title: Hidden Technical Debt in Machine Learning Systems -Abstract (p. 1): +URL Source: https://papers.nips.cc/paper_files/paper/2015/file/86df7dcfd896fcaf2674f757a2463eba-Paper.pdf -> Machine learning offers a fantastically powerful toolkit for building useful complex prediction systems quickly. This paper argues it is dangerous to think of these quick wins as coming for free. Using the software engineering framework of *technical debt*, we find it is common to incur massive ongoing maintenance costs in real-world ML systems. We explore several ML-specific risk factors to account for in system design. These include boundary erosion, entanglement, hidden feedback loops, undeclared consumers, data dependencies, configuration issues, changes in the external world, and a variety of system-level anti-patterns. +Number of Pages: 9 -Section 2, "Complex Models Erode Boundaries" — Entanglement (p. 2), the CACE principle: +Markdown Content: +# Hidden Technical Debt in Machine Learning Systems -> **Entanglement.** Machine learning systems mix signals together, entangling them and making isolation of improvements impossible. For instance, consider a system that uses features x1, ...xn in a model. If we change the input distribution of values in x1, the importance, weights, or use of the remaining n − 1 features may all change. This is true whether the model is retrained fully in a batch style or allowed to adapt in an online fashion. Adding a new feature xn+1 can cause similar changes, as can removing any feature xj. No inputs are ever really independent. We refer to this here as the CACE principle: Changing Anything Changes Everything. CACE applies not only to input signals, but also to hyper-parameters, learning settings, sampling methods, convergence thresholds, data selection, and essentially every other possible tweak. +D. Sculley, Gary Holt, Daniel Golovin, Eugene Davydov, Todd Phillips -Same section, the ensemble caveat (p. 2): +{dsculley,gholt,dgg,edavydov,toddphillips }@google.com -> One possible mitigation strategy is to isolate models and serve ensembles. [...] However, in many cases ensembles work well because the errors in the component models are uncorrelated. Relying on the combination creates a strong entanglement: improving an individual component model may actually make the system accuracy worse if the remaining errors are more strongly correlated with the other components. +Google, Inc. + +Dietmar Ebner, Vinay Chaudhary, Michael Young, Jean-Franc ¸ois Crespo, Dan Dennison + +{ebner,vchaudhary,mwyoung,jfcrespo,dennison }@google.com + +Google, Inc. + +# Abstract + +Machine learning offers a fantastically powerful toolkit for building useful com-plex prediction systems quickly. This paper argues it is dangerous to think of these quick wins as coming for free. Using the software engineering framework of technical debt , we find it is common to incur massive ongoing maintenance costs in real-world ML systems. We explore several ML-specific risk factors to account for in system design. These include boundary erosion, entanglement, hidden feedback loops, undeclared consumers, data dependencies, configuration issues, changes in the external world, and a variety of system-level anti-patterns. + +# 1 Introduction + +As the machine learning (ML) community continues to accumulate years of experience with live systems, a wide-spread and uncomfortable trend has emerged: developing and deploying ML sys-tems is relatively fast and cheap, but maintaining them over time is difficult and expensive. This dichotomy can be understood through the lens of technical debt , a metaphor introduced by Ward Cunningham in 1992 to help reason about the long term costs incurred by moving quickly in software engineering. As with fiscal debt, there are often sound strategic reasons to take on technical debt. Not all debt is bad, but all debt needs to be serviced. Technical debt may be paid down by refactoring code, improving unit tests, deleting dead code, reducing dependencies, tightening APIs, and improving documentation [8]. The goal is not to add new functionality, but to enable future improvements, reduce errors, and improve maintainability. Deferring such payments results in compounding costs. Hidden debt is dangerous because it compounds silently. In this paper, we argue that ML systems have a special capacity for incurring technical debt, because they have all of the maintenance problems of traditional code plus an additional set of ML-specific issues. This debt may be difficult to detect because it exists at the system level rather than the code level. Traditional abstractions and boundaries may be subtly corrupted or invalidated by the fact that data influences ML system behavior. Typical methods for paying down code level technical debt are not sufficient to address ML-specific technical debt at the system level. This paper does not offer novel ML algorithms, but instead seeks to increase the community’s aware-ness of the difficult tradeoffs that must be considered in practice over the long term. We focus on system-level interactions and interfaces as an area where ML technical debt may rapidly accumulate. At a system-level, an ML model may silently erode abstraction boundaries. The tempting re-use or chaining of input signals may unintentionally couple otherwise disjoint systems. ML packages may be treated as black boxes, resulting in large masses of “glue code” or calibration layers that can lock in assumptions. Changes in the external world may influence system behavior in unintended ways. Even monitoring ML system behavior may prove difficult without careful design. 12 Complex Models Erode Boundaries + +Traditional software engineering practice has shown that strong abstraction boundaries using en-capsulation and modular design help create maintainable code in which it is easy to make isolated changes and improvements. Strict abstraction boundaries help express the invariants and logical consistency of the information inputs and outputs from an given component [8]. Unfortunately, it is difficult to enforce strict abstraction boundaries for machine learning systems by prescribing specific intended behavior. Indeed, ML is required in exactly those cases when the desired behavior cannot be effectively expressed in software logic without dependency on external data . The real world does not fit into tidy encapsulation. Here we examine several ways that the resulting erosion of boundaries may significantly increase technical debt in ML systems. + +Entanglement. Machine learning systems mix signals together, entangling them and making iso-lation of improvements impossible. For instance, consider a system that uses features x1, ... xn in a model. If we change the input distribution of values in x1, the importance, weights, or use of the remaining n − 1 features may all change. This is true whether the model is retrained fully in a batch style or allowed to adapt in an online fashion. Adding a new feature xn+1 can cause similar changes, as can removing any feature xj . No inputs are ever really independent. We refer to this here as the CACE principle: Changing Anything Changes Everything. CACE applies not only to input signals, but also to hyper-parameters, learning settings, sampling methods, convergence thresholds, data selection, and essentially every other possible tweak. One possible mitigation strategy is to isolate models and serve ensembles. This approach is useful in situations in which sub-problems decompose naturally such as in disjoint multi-class settings like [14]. However, in many cases ensembles work well because the errors in the component models are uncorrelated. Relying on the combination creates a strong entanglement: improving an individual component model may actually make the system accuracy worse if the remaining errors are more strongly correlated with the other components. A second possible strategy is to focus on detecting changes in prediction behavior as they occur. One such method was proposed in [12], in which a high-dimensional visualization tool was used to allow researchers to quickly see effects across many dimensions and slicings. Metrics that operate on a slice-by-slice basis may also be extremely useful. + +Correction Cascades. There are often situations in which model ma for problem A exists, but a solution for a slightly different problem A′ is required. In this case, it can be tempting to learn a model m′ + +> a + +that takes ma as input and learns a small correction as a fast way to solve the problem. However, this correction model has created a new system dependency on ma, making it significantly more expensive to analyze improvements to that model in the future. The cost increases when correction models are cascaded, with a model for problem A′′ learned on top of m′ + +> a + +, and so on, for several slightly different test distributions. Once in place, a correction cascade can create an improvement deadlock, as improving the accuracy of any individual component actually leads to system-level detriments. Mitigation strategies are to augment ma to learn the corrections directly within the same model by adding features to distinguish among the cases, or to accept the cost of creating a separate model for A′. + +Undeclared Consumers. Oftentimes, a prediction from a machine learning model ma is made widely accessible, either at runtime or by writing to files or logs that may later be consumed by other systems. Without access controls, some of these consumers may be undeclared , silently using the output of a given model as an input to another system. In more classical software engineering, these issues are referred to as visibility debt [13]. Undeclared consumers are expensive at best and dangerous at worst, because they create a hidden tight coupling of model ma to other parts of the stack. Changes to ma will very likely impact these other parts, potentially in ways that are unintended, poorly understood, and detrimental. In practice, this tight coupling can radically increase the cost and difficulty of making any changes to ma at all, even if they are improvements. Furthermore, undeclared consumers may create hidden feedback loops, which are described more in detail in section 4. 2Undeclared consumers may be difficult to detect unless the system is specifically designed to guard against this case, for example with access restrictions or strict service-level agreements (SLAs). In the absence of barriers, engineers will naturally use the most convenient signal at hand, especially when working against deadline pressures. + +# 3 Data Dependencies Cost More than Code Dependencies + +In [13], dependency debt is noted as a key contributor to code complexity and technical debt in classical software engineering settings. We have found that data dependencies in ML systems carry a similar capacity for building debt, but may be more difficult to detect. Code dependencies can be identified via static analysis by compilers and linkers. Without similar tooling for data dependencies, it can be inappropriately easy to build large data dependency chains that can be difficult to untangle. + +Unstable Data Dependencies. To move quickly, it is often convenient to consume signals as input features that are produced by other systems. However, some input signals are unstable , meaning that they qualitatively or quantitatively change behavior over time. This can happen implicitly, when the input signal comes from another machine learning model itself that updates over time, or a data-dependent lookup table, such as for computing TF/IDF scores or semantic mappings. It can also happen explicitly, when the engineering ownership of the input signal is separate from the engineering ownership of the model that consumes it. In such cases, updates to the input signal may be made at any time. This is dangerous because even “improvements” to input signals may have arbitrary detrimental effects in the consuming system that are costly to diagnose and address. For example, consider the case in which an input signal was previously mis-calibrated. The model consuming it likely fit to these mis-calibrations, and a silent update that corrects the signal will have sudden ramifications for the model. One common mitigation strategy for unstable data dependencies is to create a versioned copy of a given signal. For example, rather than allowing a semantic mapping of words to topic clusters to change over time, it might be reasonable to create a frozen version of this mapping and use it until such a time as an updated version has been fully vetted. Versioning carries its own costs, however, such as potential staleness and the cost to maintain multiple versions of the same signal over time. + +Underutilized Data Dependencies. In code, underutilized dependencies are packages that are mostly unneeded [13]. Similarly, underutilized data dependencies are input signals that provide little incremental modeling benefit. These can make an ML system unnecessarily vulnerable to change, sometimes catastrophically so, even though they could be removed with no detriment. As an example, suppose that to ease the transition from an old product numbering scheme to new product numbers, both schemes are left in the system as features. New products get only a new number, but old products may have both and the model continues to rely on the old numbers for some products. A year later, the code that stops populating the database with the old numbers is deleted. This will not be a good day for the maintainers of the ML system. Underutilized data dependencies can creep into a model in several ways. + +• Legacy Features. The most common case is that a feature F is included in a model early in its development. Over time, F is made redundant by new features but this goes undetected. + +• Bundled Features. Sometimes, a group of features is evaluated and found to be beneficial. Because of deadline pressures or similar effects, all the features in the bundle are added to the model together, possibly including features that add little or no value. + +• ǫ-Features. As machine learning researchers, it is tempting to improve model accuracy even when the accuracy gain is very small or when the complexity overhead might be high. + +• Correlated Features. Often two features are strongly correlated, but one is more directly causal. Many ML methods have difficulty detecting this and credit the two features equally, or may even pick the non-causal one. This results in brittleness if world behavior later changes the correlations. Underutilized dependencies can be detected via exhaustive leave-one-feature-out evaluations. These should be run regularly to identify and remove unnecessary features. 3Figure 1: Only a small fraction of real-world ML systems is composed of the ML code, as shown by the small black box in the middle. The required surrounding infrastructure is vast and complex. + +Static Analysis of Data Dependencies. In traditional code, compilers and build systems perform static analysis of dependency graphs. Tools for static analysis of data dependencies are far less common, but are essential for error checking, tracking down consumers, and enforcing migration and updates. One such tool is the automated feature management system described in [12], which enables data sources and features to be annotated. Automated checks can then be run to ensure that all dependencies have the appropriate annotations, and dependency trees can be fully resolved. This kind of tooling can make migration and deletion much safer in practice. + +# 4 Feedback Loops + +One of the key features of live ML systems is that they often end up influencing their own behavior if they update over time. This leads to a form of analysis debt , in which it is difficult to predict the behavior of a given model before it is released. These feedback loops can take different forms, but they are all more difficult to detect and address if they occur gradually over time, as may be the case when models are updated infrequently. + +Direct Feedback Loops. A model may directly influence the selection of its own future training data. It is common practice to use standard supervised algorithms, although the theoretically correct solution would be to use bandit algorithms. The problem here is that bandit algorithms (such as contextual bandits [9]) do not necessarily scale well to the size of action spaces typically required for real-world problems. It is possible to mitigate these effects by using some amount of randomization [3], or by isolating certain parts of data from being influenced by a given model. + +Hidden Feedback Loops. Direct feedback loops are costly to analyze, but at least they pose a statistical challenge that ML researchers may find natural to investigate [3]. A more difficult case is + +hidden feedback loops, in which two systems influence each other indirectly through the world. One example of this may be if two systems independently determine facets of a web page, such as one selecting products to show and another selecting related reviews. Improving one system may lead to changes in behavior in the other, as users begin clicking more or less on the other components in reaction to the changes. Note that these hidden loops may exist between completely disjoint systems. Consider the case of two stock-market prediction models from two different investment companies. Improvements (or, more scarily, bugs) in one may influence the bidding and buying behavior of the other. + +# 5 ML-System Anti-Patterns + +It may be surprising to the academic community to know that only a tiny fraction of the code in many ML systems is actually devoted to learning or prediction – see Figure 1. In the language of Lin and Ryaboy, much of the remainder may be described as “plumbing” [11]. It is unfortunately common for systems that incorporate machine learning methods to end up with high-debt design patterns. In this section, we examine several system-design anti-patterns [4] that can surface in machine learning systems and which should be avoided or refactored where possible. 4Glue Code. ML researchers tend to develop general purpose solutions as self-contained packages. A wide variety of these are available as open-source packages at places like mloss.org , or from in-house code, proprietary packages, and cloud-based platforms. Using generic packages often results in a glue code system design pattern, in which a massive amount of supporting code is written to get data into and out of general-purpose packages. Glue code is costly in the long term because it tends to freeze a system to the peculiarities of a specific package; testing alternatives may become prohibitively expensive. In this way, using a generic package can inhibit improvements, because it makes it harder to take advantage of domain-specific properties or to tweak the objective function to achieve a domain-specific goal. Because a mature system might end up being (at most) 5% machine learning code and (at least) 95% glue code, it may be less costly to create a clean native solution rather than re-use a generic package. An important strategy for combating glue-code is to wrap black-box packages into common API’s. This allows supporting infrastructure to be more reusable and reduces the cost of changing packages. + +Pipeline Jungles. As a special case of glue code, pipeline jungles often appear in data prepara-tion. These can evolve organically, as new signals are identified and new information sources added incrementally. Without care, the resulting system for preparing data in an ML-friendly format may become a jungle of scrapes, joins, and sampling steps, often with intermediate files output. Man-aging these pipelines, detecting errors and recovering from failures are all difficult and costly [1]. Testing such pipelines often requires expensive end-to-end integration tests. All of this adds to technical debt of a system and makes further innovation more costly. Pipeline jungles can only be avoided by thinking holistically about data collection and feature ex-traction. The clean-slate approach of scrapping a pipeline jungle and redesigning from the ground up is indeed a major investment of engineering effort, but one that can dramatically reduce ongoing costs and speed further innovation. Glue code and pipeline jungles are symptomatic of integration issues that may have a root cause in overly separated “research” and “engineering” roles. When ML packages are developed in an ivory-tower setting, the result may appear like black boxes to the teams that employ them in practice. A hybrid research approach where engineers and researchers are embedded together on the same teams (and indeed, are often the same people) can help reduce this source of friction significantly [16]. + +Dead Experimental Codepaths. A common consequence of glue code or pipeline jungles is that it becomes increasingly attractive in the short term to perform experiments with alternative methods by implementing experimental codepaths as conditional branches within the main production code. For any individual change, the cost of experimenting in this manner is relatively low—none of the surrounding infrastructure needs to be reworked. However, over time, these accumulated codepaths can create a growing debt due to the increasing difficulties of maintaining backward compatibility and an exponential increase in cyclomatic complexity. Testing all possible interactions between codepaths becomes difficult or impossible. A famous example of the dangers here was Knight Capital’s system losing $465 million in 45 minutes, apparently because of unexpected behavior from obsolete experimental codepaths [15]. As with the case of dead flags in traditional software [13], it is often beneficial to periodically re-examine each experimental branch to see what can be ripped out. Often only a small subset of the possible branches is actually used; many others may have been tested once and abandoned. + +Abstraction Debt. The above issues highlight the fact that there is a distinct lack of strong ab-stractions to support ML systems. Zheng recently made a compelling comparison of the state ML abstractions to the state of database technology [17], making the point that nothing in the machine learning literature comes close to the success of the relational database as a basic abstraction. What is the right interface to describe a stream of data, or a model, or a prediction? For distributed learning in particular, there remains a lack of widely accepted abstractions. It could be argued that the widespread use of Map-Reduce in machine learning was driven by the void of strong distributed learning abstractions. Indeed, one of the few areas of broad agreement in recent years appears to be that Map-Reduce is a poor abstraction for iterative ML algorithms. 5The parameter-server abstraction seems much more robust, but there are multiple competing speci-fications of this basic idea [5, 10]. The lack of standard abstractions makes it all too easy to blur the lines between components. + +Common Smells. In software engineering, a design smell may indicate an underlying problem in a component or system [7]. We identify a few ML system smells, not hard-and-fast rules, but as subjective indicators. + +• Plain-Old-Data Type Smell. The rich information used and produced by ML systems is all to often encoded with plain data types like raw floats and integers. In a robust system, a model parameter should know if it is a log-odds multiplier or a decision threshold, and a prediction should know various pieces of information about the model that produced it and how it should be consumed. + +• Multiple-Language Smell. It is often tempting to write a particular piece of a system in a given language, especially when that language has a convenient library or syntax for the task at hand. However, using multiple languages often increases the cost of effective testing and can increase the difficulty of transferring ownership to other individuals. + +• Prototype Smell. It is convenient to test new ideas in small scale via prototypes. How-ever, regularly relying on a prototyping environment may be an indicator that the full-scale system is brittle, difficult to change, or could benefit from improved abstractions and inter-faces. Maintaining a prototyping environment carries its own cost, and there is a significant danger that time pressures may encourage a prototyping system to be used as a production solution. Additionally, results found at small scale rarely reflect the reality at full scale. + +# 6 Configuration Debt + +Another potentially surprising area where debt can accumulate is in the configuration of machine learning systems. Any large system has a wide range of configurable options, including which features are used, how data is selected, a wide variety of algorithm-specific learning settings, poten-tial pre- or post-processing, verification methods, etc. We have observed that both researchers and engineers may treat configuration (and extension of configuration) as an afterthought. Indeed, veri-fication or testing of configurations may not even be seen as important. In a mature system which is being actively developed, the number of lines of configuration can far exceed the number of lines of the traditional code. Each configuration line has a potential for mistakes. Consider the following examples. Feature A was incorrectly logged from 9/14 to 9/17. Feature B is not available on data before 10/7. The code used to compute feature C has to change for data before and after 11/1 because of changes to the logging format. Feature D is not available in production, so a substitute features D′ and D′′ must be used when querying the model in a live setting. If feature + +Z is used, then jobs for training must be given extra memory due to lookup tables or they will train inefficiently. Feature Q precludes the use of feature R because of latency constraints. All this messiness makes configuration hard to modify correctly, and hard to reason about. How-ever, mistakes in configuration can be costly, leading to serious loss of time, waste of computing resources, or production issues. This leads us to articulate the following principles of good configu-ration systems: + +• It should be easy to specify a configuration as a small change from a previous configuration. + +• It should be hard to make manual errors, omissions, or oversights. + +• It should be easy to see, visually, the difference in configuration between two models. + +• It should be easy to automatically assert and verify basic facts about the configuration: number of features used, transitive closure of data dependencies, etc. + +• It should be possible to detect unused or redundant settings. + +• Configurations should undergo a full code review and be checked into a repository. 67 Dealing with Changes in the External World + +One of the things that makes ML systems so fascinating is that they often interact directly with the external world. Experience has shown that the external world is rarely stable. This background rate of change creates ongoing maintenance cost. + +Fixed Thresholds in Dynamic Systems. It is often necessary to pick a decision threshold for a given model to perform some action: to predict true or false, to mark an email as spam or not spam, to show or not show a given ad. One classic approach in machine learning is to choose a threshold from a set of possible thresholds, in order to get good tradeoffs on certain metrics, such as precision and recall. However, such thresholds are often manually set. Thus if a model updates on new data, the old manually set threshold may be invalid. Manually updating many thresholds across many models is time-consuming and brittle. One mitigation strategy for this kind of problem appears in [14], in which thresholds are learned via simple evaluation on heldout validation data. + +Monitoring and Testing. Unit testing of individual components and end-to-end tests of running systems are valuable, but in the face of a changing world such tests are not sufficient to provide evidence that a system is working as intended. Comprehensive live monitoring of system behavior in real time combined with automated response is critical for long-term system reliability. The key question is: what to monitor? Testable invariants are not always obvious given that many ML systems are intended to adapt over time. We offer the following starting points. + +• Prediction Bias. In a system that is working as intended, it should usually be the case that the distribution of predicted labels is equal to the distribution of observed labels. This is by no means a comprehensive test, as it can be met by a null model that simply predicts average values of label occurrences without regard to the input features. However, it is a surprisingly useful diagnostic, and changes in metrics such as this are often indicative of an issue that requires attention. For example, this method can help to detect cases in which the world behavior suddenly changes, making training distributions drawn from historical data no longer reflective of current reality. Slicing prediction bias by various dimensions isolate issues quickly, and can also be used for automated alerting. + +• Action Limits. In systems that are used to take actions in the real world, such as bidding on items or marking messages as spam, it can be useful to set and enforce action limits as a sanity check. These limits should be broad enough not to trigger spuriously. If the system hits a limit for a given action, automated alerts should fire and trigger manual intervention or investigation. + +• Up-Stream Producers. Data is often fed through to a learning system from various up-stream producers. These up-stream processes should be thoroughly monitored, tested, and routinely meet a service level objective that takes the downstream ML system needs into account. Further any up-stream alerts must be propagated to the control plane of an ML system to ensure its accuracy. Similarly, any failure of the ML system to meet established service level objectives be also propagated down-stream to all consumers, and directly to their control planes if at all possible. Because external changes occur in real-time, response must also occur in real-time as well. Relying on human intervention in response to alert pages is one strategy, but can be brittle for time-sensitive issues. Creating systems to that allow automated response without direct human intervention is often well worth the investment. + +# 8 Other Areas of ML-related Debt + +We now briefly highlight some additional areas where ML-related technical debt may accrue. + +Data Testing Debt. If data replaces code in ML systems, and code should be tested, then it seems clear that some amount of testing of input data is critical to a well-functioning system. Basic sanity checks are useful, as more sophisticated tests that monitor changes in input distributions. 7Reproducibility Debt. As scientists, it is important that we can re-run experiments and get similar results, but designing real-world systems to allow for strict reproducibility is a task made difficult by randomized algorithms, non-determinism inherent in parallel learning, reliance on initial conditions, and interactions with the external world. + +Process Management Debt. Most of the use cases described in this paper have talked about the cost of maintaining a single model, but mature systems may have dozens or hundreds of models running simultaneously [14, 6]. This raises a wide range of important problems, including the problem of updating many configurations for many similar models safely and automatically, how to manage and assign resources among models with different business priorities, and how to visualize and detect blockages in the flow of data in a production pipeline. Developing tooling to aid recovery from production incidents is also critical. An important system-level smell to avoid are common processes with many manual steps. + +Cultural Debt. There is sometimes a hard line between ML research and engineering, but this can be counter-productive for long-term system health. It is important to create team cultures that reward deletion of features, reduction of complexity, improvements in reproducibility, stability, and monitoring to the same degree that improvements in accuracy are valued. In our experience, this is most likely to occur within heterogeneous teams with strengths in both ML research and engineering. + +# 9 Conclusions: Measuring Debt and Paying it Off + +Technical debt is a useful metaphor, but it unfortunately does not provide a strict metric that can be tracked over time. How are we to measure technical debt in a system, or to assess the full cost of this debt? Simply noting that a team is still able to move quickly is not in itself evidence of low debt or good practices, since the full cost of debt becomes apparent only over time. Indeed, moving quickly often introduces technical debt. A few useful questions to consider are: + +• How easily can an entirely new algorithmic approach be tested at full scale? + +• What is the transitive closure of all data dependencies? + +• How precisely can the impact of a new change to the system be measured? + +• Does improving one model or signal degrade others? + +• How quickly can new members of the team be brought up to speed? We hope that this paper may serve to encourage additional development in the areas of maintainable ML, including better abstractions, testing methodologies, and design patterns. Perhaps the most important insight to be gained is that technical debt is an issue that engineers and researchers both need to be aware of. Research solutions that provide a tiny accuracy benefit at the cost of massive increases in system complexity are rarely wise practice. Even the addition of one or two seemingly innocuous data dependencies can slow further progress. Paying down ML-related technical debt requires a specific commitment, which can often only be achieved by a shift in team culture. Recognizing, prioritizing, and rewarding this effort is important for the long term health of successful ML teams. + +Acknowledgments + +This paper owes much to the important lessons learned day to day in a culture that values both innovative ML research and strong engineering practice. Many colleagues have helped shape our thoughts here, and the benefit of accumulated folk wisdom cannot be overstated. We would like to specifically recognize the following: Roberto Bayardo, Luis Cobo, Sharat Chikkerur, Jeff Dean, Philip Henderson, Arnar Mar Hrafnkelsson, Ankur Jain, Joe Kovac, Jeremy Kubica, H. Brendan McMahan, Satyaki Mahalanabis, Lan Nie, Michael Pohl, Abdul Salem, Sajid Siddiqi, Ricky Shan, Alan Skelly, Cory Williams, and Andrew Young. A short version of this paper was presented at the SE4ML workshop in 2014 in Montreal, Canada. 8References + +[1] R. Ananthanarayanan, V. Basker, S. Das, A. Gupta, H. Jiang, T. Qiu, A. Reznichenko, D. Ryabkov, M. Singh, and S. Venkataraman. Photon: Fault-tolerant and scalable joining of continuous data streams. In SIGMOD ’13: Proceedings of the 2013 international conference on Management of data , pages 577– 588, New York, NY, USA, 2013. [2] A. Anonymous. Machine learning: The high-interest credit card of technical debt. SE4ML: Software Engineering for Machine Learning (NIPS 2014 Workshop) .[3] L. Bottou, J. Peters, J. Qui˜ nonero Candela, D. X. Charles, D. M. Chickering, E. Portugaly, D. Ray, P. Simard, and E. Snelson. Counterfactual reasoning and learning systems: The example of computational advertising. Journal of Machine Learning Research , 14(Nov), 2013. [4] W. J. Brown, H. W. McCormick, T. J. Mowbray, and R. C. Malveau. Antipatterns: refactoring software, architectures, and projects in crisis. 1998. [5] T. M. Chilimbi, Y. Suzue, J. Apacible, and K. Kalyanaraman. Project adam: Building an efficient and scalable deep learning training system. In 11th USENIX Symposium on Operating Systems Design and Implementation, OSDI ’14, Broomfield, CO, USA, October 6-8, 2014. , pages 571–582, 2014. [6] B. Dalessandro, D. Chen, T. Raeder, C. Perlich, M. Han Williams, and F. Provost. Scalable hands-free transfer learning for online advertising. In Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining , pages 1573–1582. ACM, 2014. [7] M. Fowler. Code smells. http://http://martinfowler.com/bliki/CodeSmell.html .[8] M. Fowler. Refactoring: improving the design of existing code . Pearson Education India, 1999. [9] J. Langford and T. Zhang. The epoch-greedy algorithm for multi-armed bandits with side information. In + +Advances in neural information processing systems , pages 817–824, 2008. [10] M. Li, D. G. Andersen, J. W. Park, A. J. Smola, A. Ahmed, V. Josifovski, J. Long, E. J. Shekita, and B. Su. Scaling distributed machine learning with the parameter server. In 11th USENIX Symposium on Operating Systems Design and Implementation, OSDI ’14, Broomfield, CO, USA, October 6-8, 2014. ,pages 583–598, 2014. [11] J. Lin and D. Ryaboy. Scaling big data mining infrastructure: the twitter experience. ACM SIGKDD Explorations Newsletter , 14(2):6–19, 2013. [12] H. B. McMahan, G. Holt, D. Sculley, M. Young, D. Ebner, J. Grady, L. Nie, T. Phillips, E. Davydov, D. Golovin, S. Chikkerur, D. Liu, M. Wattenberg, A. M. Hrafnkelsson, T. Boulos, and J. Kubica. Ad click prediction: a view from the trenches. In The 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD 2013, Chicago, IL, USA, August 11-14, 2013 , 2013. [13] J. D. Morgenthaler, M. Gridnev, R. Sauciuc, and S. Bhansali. Searching for build debt: Experiences managing technical debt at google. In Proceedings of the Third International Workshop on Managing Technical Debt , 2012. [14] D. Sculley, M. E. Otey, M. Pohl, B. Spitznagel, J. Hainsworth, and Y. Zhou. Detecting adversarial adver-tisements in the wild. In Proceedings of the 17th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Diego, CA, USA, August 21-24, 2011 , 2011. [15] Securities and E. Commission. SEC Charges Knight Capital With Violations of Market Access Rule , 2013. [16] A. Spector, P. Norvig, and S. Petrov. Google’s hybrid approach to research. Communications of the ACM ,55 Issue 7, 2012. [17] A. Zheng. The challenges of building machine learning tools for the masses. SE4ML: Software Engineer-ing for Machine Learning (NIPS 2014 Workshop) . + +9