A Taxonomy of Automatic Differentiation Pitfalls

Jan Hückelheim; Harshitha Menon; William Moses; Bruce Christianson; Paul Hovland; Laurent Hascoët

doi:10.1002/widm.1555

A Taxonomy of Automatic Differentiation Pitfalls

Jan Hückelheim, Harshitha Menon, William Moses, Bruce Christianson, Paul Hovland, Laurent Hascoët

Research output: Contribution to journal › Article › peer-review

25 Downloads (Pure)

Abstract

Automatic differentiation is a popular technique for computing derivatives of computer programs. While automatic differentiation has been successfully used in countless engineering, science and machine learning applications, it can sometimes nevertheless produce surprising results. In this paper we categorize problematic usages of automatic differentiation, and illustrate each category with examples such as chaos, time-averages, discretizations, fixed-point loops, lookup tables, linear solvers, and probabilistic programs, in the hope that readers may more easily avoid or detect such pitfalls. We also review debugging techniques and their effectiveness in these situations.

Original language	English
Number of pages	12
Journal	WIREs: Data Mining and Knowledge Discovery
Early online date	2 Sept 2024
DOIs	https://doi.org/10.1002/widm.1555
Publication status	E-pub ahead of print - 2 Sept 2024

Keywords

Autodiff, Automatic Differentiation, Backpropagation

Access to Document

10.1002/widm.1555Licence: CC BY

Final_Version
© 2024 UChicago Argonne, LLC and The Author(s). This is an open access article under the terms of the Creative Commons Attribution License: https://creativecommons.org/licenses/by/4.0/
Final published version, 1.85 MBLicence: CC BY

Cite this

@article{3f67c02154e245aeb95880b8897a0c3c,

title = "A Taxonomy of Automatic Differentiation Pitfalls",

abstract = "Automatic differentiation is a popular technique for computing derivatives of computer programs. While automatic differentiation has been successfully used in countless engineering, science and machine learning applications, it can sometimes nevertheless produce surprising results. In this paper we categorize problematic usages of automatic differentiation, and illustrate each category with examples such as chaos, time-averages, discretizations, fixed-point loops, lookup tables, linear solvers, and probabilistic programs, in the hope that readers may more easily avoid or detect such pitfalls. We also review debugging techniques and their effectiveness in these situations.",

keywords = "Autodiff, Automatic Differentiation, Backpropagation",

author = "Jan H{\"u}ckelheim and Harshitha Menon and William Moses and Bruce Christianson and Paul Hovland and Laurent Hasco{\"e}t",

note = "{\textcopyright} 2024 UChicago Argonne, LLC and The Author(s). This is an open access article under the terms of the Creative Commons Attribution License, https://creativecommons.org/licenses/by/4.0/",

year = "2024",

month = sep,

day = "2",

doi = "10.1002/widm.1555",

language = "English",

journal = "WIREs: Data Mining and Knowledge Discovery",

issn = "1942-4787",

publisher = "Wiley",

}

TY - JOUR

T1 - A Taxonomy of Automatic Differentiation Pitfalls

AU - Hückelheim, Jan

AU - Menon, Harshitha

AU - Moses, William

AU - Christianson, Bruce

AU - Hovland, Paul

AU - Hascoët, Laurent

PY - 2024/9/2

Y1 - 2024/9/2

N2 - Automatic differentiation is a popular technique for computing derivatives of computer programs. While automatic differentiation has been successfully used in countless engineering, science and machine learning applications, it can sometimes nevertheless produce surprising results. In this paper we categorize problematic usages of automatic differentiation, and illustrate each category with examples such as chaos, time-averages, discretizations, fixed-point loops, lookup tables, linear solvers, and probabilistic programs, in the hope that readers may more easily avoid or detect such pitfalls. We also review debugging techniques and their effectiveness in these situations.

AB - Automatic differentiation is a popular technique for computing derivatives of computer programs. While automatic differentiation has been successfully used in countless engineering, science and machine learning applications, it can sometimes nevertheless produce surprising results. In this paper we categorize problematic usages of automatic differentiation, and illustrate each category with examples such as chaos, time-averages, discretizations, fixed-point loops, lookup tables, linear solvers, and probabilistic programs, in the hope that readers may more easily avoid or detect such pitfalls. We also review debugging techniques and their effectiveness in these situations.

KW - Autodiff, Automatic Differentiation, Backpropagation

U2 - 10.1002/widm.1555

DO - 10.1002/widm.1555

M3 - Article

SN - 1942-4787

JO - WIREs: Data Mining and Knowledge Discovery

JF - WIREs: Data Mining and Knowledge Discovery

ER -