Automatic Differentiation

​
x
0
2
y
0
3
a0
1
b0
2
c0
3
enter var
x
y
x + e
x + h e
y + k e
con
a
b
c
op
-
+
*
/
1/x
ln
e^x
y^x
log
10^x
x^2
sin
cos
tan
x^(1/2)
asin
acos
atan
x ⟷ y
chs
pop
clear
generic ops
f
g
F
G
memory
sto1
sto2
sto3
rcl1
rcl2
rcl3
lastx
list of clicked items, expression pile, value pile, at
x
0
= 2, or (
x
0
,
y
0
) = (2,3)and (
a
0
,
b
0
,
c
0
) = (1,2,3)
{}
​
​
sto1
sto2
sto3
This is a dual number expression RPN (reverse Polish notation) calculator for automatic differentiation, modeled on an old-style scientific calculator. Dual numbers are numbers of the form
a+bϵ
, where
a
,
b
are real and
2
ϵ
=0
. (
a+bϵ
is displayed with
a
and
b
in adjacent boxes. Use the calculator to calculate an arithmetic expression in
x+ϵ
and reals
a
,
b
, and
c
). Clicking a value puts the value on top of a pile. Clicking an "op" applies the "op" to the top member(s) of the pile as argument(s), and replaces them on the pile with its value. This is the RPN method of evaluation. The list of buttons clicked will be the reverse Polish notation for the expression. If the result is
f(x+ϵ)
for a function
f
, its value will be
f(x)+f'(x)ϵ
. Thus,
f'(x)
is calculated "automatically". Moreover, this will be the result if
f
is any function computable by the calculator. In fact,
f(x+hϵ)=f(x)+hf'(x)ϵ
. It is as if
f(x+hϵ)
were expanded in a Maclaurin series in
ϵ
, since
2
ϵ
=0
.
Click "x + e", "x + e", "*" to get
2
x
+2xϵ
.
Click "x + e", "x + e", *, "a", +, "x + e", "*" to get
(
2
x
+a)x+(3
2
x
+a)ϵ
.
Click "x + e", "x^2", "x + e", "1/x", "sin", "*" to get
2
x
sin
1
x
+2xsin
1
x
-cos
1
x
ϵ
.
Click "x + h e", "cos", "x^2" to get
2
cos
(x)-2hsin(x)cos(x)ϵ
.
Calculating an expression
f(x+hϵ,y+kϵ)
in dual numbers
x+hϵ
and
y+hϵ
results in
f(x,y)+(h
f
1
(x,y)+k
f
2
(x,y))ϵ
.
For example, to compute the partial derivatives of
x
2
y
2
x
+
2
y
, click "x + h e", "y + k e", "x^2", "*", "x + h e", "x^2", "y + k e", "x^2", "+", "/". The partial derivatives are the coefficients of
h
and
k
in the second box.
Built-in generic functions
f
and
g
of one variable and
F
and
G
of two variables can be used to derive general differentiation formulas.
For example, compute the formula for the derivative of
f(x)
g(x)
by clicking "x + e", "f", "x + e", "g", "/" and for
f(ax)
by clicking "a", "x + e", "*", "f".
The expression pile contains symbolic expressions and the value pile contains these expressions evaluated at
x=
x
0
,
y=
y
0
,
a=
a
0
,
b=
b
0
, and
c=
c
0
. (To get decimal values, set an involved slider to a decimal value.) Click "clear" to start a new calculation. The calculator can also be used to calculate real expressions and their values in
x
,
y
,
a
,
b
, and
c
.

Details

The calculator works by extending built-in functions
f
of one or two real variables to duals by
f(x+hϵ)=f(x)+hf'(x)ϵ
and
f(x+hϵ,y+kϵ)=f(x,y)+(h
f
1
(x,y)+k
f
2
(x,y))ϵ
. The chain rule implies these relations hold for every computable function.
For example, if
f(x+hϵ)=f(x)+hf'(x)ϵ
and
g(x+hϵ)=g(x)+hg'(x)ϵ
, then
f(g(x)+hg'(x)ϵ)=f(g(x))+hg'(x)f'(g(x))ϵ=(f∘g)(x)+h(f∘g)'(x)ϵ
.
In Dual Number, a dual number is a number of the form
a+bϵ
, where
a
and
b
are real, and
ϵ
is a matrix such that
2
ϵ
=0
, such as
ϵ=
0
1
0
0

.
Our calculator represents the dual number
a+bϵ
as the expression
dual[a,b]
, with operations defined by
dual[a,b]+dual[c,d]=dual[a+c,b+d]
,
dual[a,b]*dual[c,d]=dual[ac,ad+bc]
,
adual[b,c]=dual[ab,ac]
, which defines negation (
a=-1
) and hence subtraction,
n
dual[a,b]
=dual[
n
a
,bn
n-1
a
]
for real
n
, which defines reciprocal (
n=-1
) and hence division,
f(dual[x,h])=dual[f(x),hf'(x)]
, for functions
f
of one variable,
f(dual[x,h],dual[y,k])=dual[f(x,y),h
f
1
(x,y)+k
f
2
(x,y)]
for functions
f
of two variables,
dual[c,d]
dual[a,b]
=exp(dual[c,d]*log(dual[a,b]))
.
These rules are implemented in Mathematica using "up values". With operations thus overloaded, the calculator is implemented as if it were defined only for reals.

References

[1] D. Piponi, "Automatic Differentiation". A Neighborhood of Infinity. (Jul 28, 2005) blog.sigfpe.com/2005/07/automatic-differentiation.html.
[2] L. B. Rall, "The Arithmetic of Differentiation," Mathematics Magazine, 59(5), 1986, pp. 275–282.
[3] R. D. Neidinger, "Automatic Differentiation and APL," The College Mathematics Journal, 20(3), 1989 pp. 238–251.
[4] R. D. Neidinger, "Introduction to Automatic Differentiation and MATLAB Obect Oriented Programming," SIAM Review, 52(3), 2010 pp. 545–563.
Also see the Wikipedia entries for Dual number and Automatic differentiation.

Permanent Citation

Roger B. Kirchner
​
​"Automatic Differentiation" from the Wolfram Demonstrations Project http://demonstrations.wolfram.com/AutomaticDifferentiation/​
​Published: September 18, 2009
© Wolfram Demonstrations Project & Contributors |Terms of Use