Software Engineer's Notes

A small change in a software system’s input can sometimes produce a large change in its output. In other cases, an input may barely affect the result at all.

Understanding these relationships is essential in artificial intelligence, scientific computing, robotics, optimization, and any software that models a complex system.

The Jacobian matrix provides a structured way to answer an important question:

How does every output of a system change when each input changes slightly?

It may appear to be an abstract mathematical concept, but the Jacobian is deeply connected to practical software development. Neural network training, backpropagation, automatic differentiation, optimization algorithms, normalizing flows, inverse kinematics, sensitivity analysis, and differentiable simulations all depend on Jacobians or operations derived from them.

What Is the Jacobian Matrix?

The ordinary derivative describes how a single output changes with respect to a single input.

For example, consider:

f(x) = x²

Its derivative is:

f'(x) = 2x

This derivative tells us how much the output changes when x changes slightly.

However, real software systems frequently have multiple inputs and multiple outputs.

Suppose we have a function:

F(x₁, x₂, ..., xₙ) = [f₁(x), f₂(x), ..., fₘ(x)]

This function maps n input values to m output values:

F: ℝⁿ → ℝᵐ

The Jacobian matrix contains the partial derivative of every output with respect to every input:

             ∂f₁/∂x₁   ∂f₁/∂x₂   ...   ∂f₁/∂xₙ
             ∂f₂/∂x₁   ∂f₂/∂x₂   ...   ∂f₂/∂xₙ
J_F(x) =        ...        ...     ...      ...
             ∂fₘ/∂x₁   ∂fₘ/∂x₂   ...   ∂fₘ/∂xₙ

For a function with n inputs and m outputs, the Jacobian normally has the shape:

m × n

Each row represents one output. Each column represents one input. JAX documentation describes the Jacobian of F: ℝⁿ → ℝᵐ as a matrix in ℝᵐˣⁿ.

The History and Roots of the Jacobian

The Jacobian is named after the German mathematician Carl Gustav Jacob Jacobi, who lived from 1804 to 1851.

Jacobi made major contributions to determinants, differential equations, mechanics, number theory, and elliptic functions. The object now called the Jacobian determinant was originally studied as a functional determinant.

The historical development was gradual. Augustin-Louis Cauchy had already investigated a form of the functional determinant in 1815. Jacobi later developed the subject extensively and published a major memoir titled De determinantibus functionalibus in 1841. His work established important relationships between functional dependence, determinants, and systems of multivariable functions.

The original motivation was not artificial intelligence. Mathematicians needed better tools for:

• Transforming coordinate systems
• Solving systems of equations
• Studying differential equations
• Determining whether variables were functionally dependent
• Understanding local invertibility
• Analyzing physical and mechanical systems

The same mathematical problems now appear inside software.

A neural network is a composition of vector-valued functions. A robot maps joint positions to physical coordinates. A simulation maps parameters to predicted behavior. A generative model transforms a simple probability distribution into a complex one.

In all these cases, the Jacobian measures how inputs influence outputs.

The connection to modern AI became especially important through automatic differentiation and backpropagation. The influential 1986 work of David Rumelhart, Geoffrey Hinton, and Ronald Williams described the use of backpropagation to adjust neural-network weights by propagating output errors backward through a network.

At a mathematical level, that backward propagation is an efficient application of the chain rule using Jacobian-related operations.

Why Do We Need the Jacobian?

A single derivative is not sufficient when a function has several inputs or several outputs.

Imagine a recommendation model that receives:

Input:
- User age
- Purchase history
- Product price
- Product category
- Time of day

It may produce:

Output:
- Probability of clicking
- Probability of purchasing
- Predicted order value

There is not one derivative that describes this system. Each output can respond differently to every input.

The Jacobian organizes all these relationships:

                         Age   History   Price   Category   Time

Click probability         ∂       ∂        ∂        ∂        ∂
Purchase probability      ∂       ∂        ∂        ∂        ∂
Predicted order value     ∂       ∂        ∂        ∂        ∂

This makes it possible to determine:

• Which inputs have the strongest influence
• Which outputs are most sensitive
• Whether small perturbations could destabilize the system
• How an error should be propagated backward
• Whether a transformation can be locally reversed
• How a probability density changes after a transformation

The Jacobian exists because complex systems require a multidimensional version of the derivative.

The Main Intuition: A Local Linear Approximation

The most useful way to understand the Jacobian is as a local linear approximation.

A nonlinear function may be complicated globally. However, when we zoom in closely around a particular point, it often behaves approximately like a linear transformation.

For a small input change Δx:

F(x + Δx) ≈ F(x) + J_F(x)Δx

The Jacobian converts a small input change into an estimated output change:

Estimated output change = Jacobian × Input change

This is important because linear systems are easier to analyze and compute than nonlinear systems.

The Jacobian allows software to temporarily treat a nonlinear system as linear near its current state.

A Simple Jacobian Example

Consider the following function:

F(x, y) = [
    x² + y,
    xy
]

It has two inputs and two outputs:

f₁(x, y) = x² + y
f₂(x, y) = xy

Calculate the partial derivatives:

∂f₁/∂x = 2x
∂f₁/∂y = 1
∂f₂/∂x = y
∂f₂/∂y = x

The Jacobian is:

J_F(x, y) = [
    [2x, 1],
    [ y, x]
]

At the point (2, 3):

J_F(2, 3) = [
    [4, 1],
    [3, 2]
]

Suppose the input changes by:

Δx = 0.01
Δy = -0.02

The Jacobian estimates the output change:

[
    [4, 1],
    [3, 2]
]
×
[
     0.01,
    -0.02
]
=
[
     0.02,
    -0.01
]

Therefore:

First output increases by approximately 0.02.
Second output decreases by approximately 0.01.

The exact output change is very close to this approximation. The approximation becomes increasingly accurate as the input change becomes smaller.

The Jacobian Matrix vs. the Jacobian Determinant

The terms Jacobian matrix and Jacobian determinant are sometimes incorrectly used interchangeably.

Jacobian matrix

The Jacobian matrix contains all first-order partial derivatives.

It can be rectangular:

m outputs × n inputs

Jacobian determinant

The determinant exists only when the Jacobian matrix is square:

Number of outputs = Number of inputs

For the previous example:

J = [
    [4, 1],
    [3, 2]
]

Its determinant is:

det(J) = (4 × 2) - (1 × 3) = 5

The absolute value of the determinant describes the local scaling of area or volume under the transformation. A determinant close to zero indicates that the transformation is locally compressing dimensions or becoming difficult to invert. A nonzero determinant is associated with local invertibility under the conditions of the inverse function theorem. Jacobian-based scaling is also central to change-of-variable calculations.

The determinant’s sign can also indicate whether the transformation preserves or reverses orientation.

Jacobian, Gradient, and Hessian

These concepts are closely related but represent different mathematical objects.

Derivative

A derivative usually refers to a scalar input and scalar output:

f: ℝ → ℝ

Gradient

A gradient describes a scalar output with multiple inputs:

f: ℝⁿ → ℝ

The gradient contains one partial derivative for each input:

∇f(x) = [
    ∂f/∂x₁,
    ∂f/∂x₂,
    ...,
    ∂f/∂xₙ
]

Depending on the convention, it may be represented as a row or column vector.

Jacobian

A Jacobian describes multiple outputs with multiple inputs:

F: ℝⁿ → ℝᵐ

It contains the first derivative of every output with respect to every input.

Hessian

The Hessian contains second-order partial derivatives of a scalar-valued function:

H_f(x) = [
    ∂²f/∂x₁²        ∂²f/∂x₁∂x₂
    ∂²f/∂x₂∂x₁     ∂²f/∂x₂²
]

A Hessian can be interpreted as the Jacobian of a gradient.

In practice:

• The gradient tells us the direction of change.
• The Jacobian describes the sensitivity of several outputs.
• The Hessian describes curvature.

How the Jacobian Supports the Chain Rule

Modern software systems are frequently built as compositions of functions:

F(x) = f₃(f₂(f₁(x)))

For example, a neural network may contain:

Input → Linear layer → Activation → Linear layer → Output

Each component has its own Jacobian.

The chain rule states that the Jacobian of the complete system is obtained by multiplying the Jacobians of its components in the correct order:

J_F = J_f₃ × J_f₂ × J_f₁

This is one of the most important reasons the Jacobian matters in AI.

A neural network can contain millions or billions of parameters. Explicitly constructing and multiplying every full Jacobian would be extremely expensive. Instead, automatic differentiation systems efficiently compute products involving Jacobians.

Jacobian-Vector Products and Vector-Jacobian Products

In large AI systems, developers usually do not need the entire Jacobian matrix.

They need the result of multiplying the Jacobian by a vector.

Jacobian-vector product

A Jacobian-vector product, or JVP, calculates:

Jv

It answers:

How does the output change if the input moves in direction v?

JVPs are associated with forward-mode automatic differentiation.

Vector-Jacobian product

A vector-Jacobian product, or VJP, calculates:

vᵀJ

It propagates information from outputs back toward inputs.

VJPs are associated with reverse-mode automatic differentiation, which is the foundation of ordinary backpropagation.

JAX provides both jvp and vjp operations, as well as jacfwd and jacrev for complete Jacobians. Its documentation explains that forward mode is generally better for Jacobians with relatively few inputs and many outputs, while reverse mode is generally better when there are many inputs and fewer outputs.

This explains why reverse-mode differentiation works well for neural-network training:

Inputs: Millions of model parameters
Output: One scalar loss

TensorFlow similarly notes that reverse mode is attractive for scalar-valued outputs with many inputs, while forward mode is useful when a function has relatively few inputs and many outputs.

Why the Jacobian Is Important in Artificial Intelligence

1. Neural network training

A neural network is a sequence of differentiable transformations.

During training, the model calculates a loss:

Loss = Difference between prediction and expected result

Backpropagation computes how that loss changes with respect to model parameters.

Mathematically, this process repeatedly applies vector-Jacobian products through the layers of the network.

Without Jacobian-related operations, gradient-based neural-network training would not be practical.

2. Input sensitivity and explainability

The Jacobian of model outputs with respect to model inputs can show how sensitive predictions are to individual features.

For a model:

prediction = model(input)

we can calculate:

∂prediction/∂input

Large derivative values indicate that small input changes may significantly affect the prediction.

This can support:

• Feature sensitivity analysis
• Saliency methods
• Model debugging
• Detection of unstable predictions
• Investigation of unexpected model behavior

A sensitivity value should not automatically be interpreted as causation. It describes local mathematical influence, not necessarily a real-world causal relationship.

3. Adversarial robustness

A model with a very large input-output Jacobian can be highly sensitive to small input changes.

Researchers have investigated Jacobian regularization as a way to encourage smoother and more stable neural-network behavior. Such approaches penalize large Jacobian norms during or after training. Research has connected this technique with improved resistance to random and adversarial input perturbations.

A simplified training objective might be:

Total loss =
    Prediction loss
    + λ × Jacobian penalty

The penalty encourages the model to avoid excessive sensitivity.

4. Normalizing flows

Normalizing flows create complex probability distributions by applying a series of invertible transformations to a simpler distribution.

When a transformation changes the geometry of a probability distribution, the density must be corrected using the Jacobian determinant.

A simplified change-of-variables expression is:

log p(y) =
    log p(x)
    - log |det(J)|

Normalizing-flow architectures are designed so that the Jacobian determinant is computationally manageable. Rezende and Mohamed’s work on variational inference with normalizing flows emphasizes invertible transformations and efficient handling of Jacobian determinants.

Jacobians are therefore central to:

• Flow-based generative models
• Density estimation
• Variational inference
• Probabilistic modeling
• Some generative AI architectures

5. Neural ordinary differential equations

Neural ordinary differential equations, or Neural ODEs, model hidden-state evolution as a continuous process:

dh/dt = f(h, t, θ)

Training these systems requires sensitivity calculations involving derivatives of the dynamics with respect to states and parameters.

The original Neural ODE work uses adjoint sensitivity methods and vector-Jacobian products to train models through differential equation solvers.

6. Optimization

Optimization algorithms use derivatives to decide how variables should change.

For a system of nonlinear equations:

F(x) = 0

Newton-style methods use the Jacobian:

J_F(x)Δx = -F(x)

The algorithm solves this local linear system and updates:

x_new = x + Δx

Jacobians appear in:

• Newton’s method
• Gauss-Newton optimization
• Nonlinear least squares
• Parameter estimation
• Inverse problems
• System identification

7. Robotics and control systems

A robot arm maps joint angles to the position and orientation of its end effector:

Joint configuration → End-effector position

The robot Jacobian maps joint velocities to end-effector velocities:

End-effector velocity = Jacobian × Joint velocity

It is used for:

• Inverse kinematics
• Motion planning
• Force control
• Trajectory optimization
• Singularity detection
• Real-time feedback control

A loss of Jacobian rank can indicate a robotic singularity in which certain movement directions become unavailable or require extremely large joint velocities.

8. Computer vision and graphics

Coordinate transformations are common in graphics and vision:

• World coordinates to camera coordinates
• Camera coordinates to image coordinates
• Geometric warping
• Lens distortion correction
• Pose estimation
• Image registration

Jacobians describe how a small movement in one coordinate system affects another.

They are also used when optimizing camera parameters, reconstructing three-dimensional scenes, or differentiating through rendering pipelines.

9. Scientific machine learning

Scientific machine learning combines physical equations with machine-learning models.

Jacobians are important for:

• Differentiable simulations
• Physics-informed neural networks
• Parameter calibration
• Solving differential equations
• Sensitivity analysis
• Surrogate modeling
• Digital twins

In these applications, developers may differentiate through an entire simulation to understand how physical outputs depend on parameters.

Benefits of Using Jacobians

Structured sensitivity analysis

The Jacobian provides one organized representation of every first-order input-output relationship.

Efficient gradient propagation

Automatic differentiation frameworks avoid explicitly building unnecessary matrices and compute JVPs or VJPs efficiently.

Better understanding of model stability

Jacobian norms, singular values, and condition numbers can reveal whether a system is excessively sensitive or nearly singular.

Support for optimization

Many numerical algorithms depend on local linear approximations derived from Jacobians.

Improved debugging

Unexpected zero, extremely large, NaN, or infinite derivatives may reveal:

• Disconnected computation graphs
• Saturated activation functions
• Incorrect custom gradients
• Numerical overflow
• Poorly scaled inputs
• Non-differentiable operations

Coordinate and probability transformations

The Jacobian determinant correctly accounts for local area, volume, and probability-density changes.

Reusable mathematical abstraction

The same concept applies to neural networks, robots, graphics systems, simulations, probability models, and optimization software.

Computing a Jacobian with PyTorch

PyTorch provides composable differentiation functions through torch.func, including jacrev and jacfwd.

import torch
from torch.func import jacrev, jacfwd
def transform(inputs: torch.Tensor) -> torch.Tensor:
    x, y = inputs
    return torch.stack(
        (
            x**2 + y,
            x * y,
        )
    )
point = torch.tensor([2.0, 3.0])
jacobian_reverse = jacrev(transform)(point)
jacobian_forward = jacfwd(transform)(point)
print(jacobian_reverse)
print(jacobian_forward)

The result is:

tensor([
    [4., 1.],
    [3., 2.]
])

Both methods calculate the same mathematical Jacobian but use different automatic differentiation strategies.

Computing a Jacobian with JAX

JAX provides jacfwd for forward-mode differentiation and jacrev for reverse-mode differentiation.

import jax
import jax.numpy as jnp
def transform(inputs):
    x, y = inputs
    return jnp.array(
        [
            x**2 + y,
            x * y,
        ]
    )
point = jnp.array([2.0, 3.0])
jacobian_forward = jax.jacfwd(transform)(point)
jacobian_reverse = jax.jacrev(transform)(point)
print(jacobian_forward)
print(jacobian_reverse)

Expected result:

[[4. 1.]
 [3. 2.]]

Computing a Jacobian with TensorFlow

TensorFlow uses tf.GradientTape to record differentiable operations. The tape can then calculate gradients or Jacobians.

import tensorflow as tf
def transform(inputs):
    x = inputs[0]
    y = inputs[1]
    return tf.stack(
        [
            x**2 + y,
            x * y,
        ]
    )
point = tf.Variable([2.0, 3.0])
with tf.GradientTape() as tape:
    output = transform(point)
jacobian = tape.jacobian(output, point)
print(jacobian)

Expected result:

[[4. 1.]
 [3. 2.]]

Key Aspects Developers Should Understand

Shape

For:

F: ℝⁿ → ℝᵐ

the Jacobian shape is usually:

m × n

Shape errors are among the most common problems in Jacobian-related code.

Evaluation point

A Jacobian is normally evaluated at a specific input.

A nonlinear function can have a different Jacobian at every point:

J_F(x₁) ≠ J_F(x₂)

Local meaning

The Jacobian describes local behavior. It does not necessarily describe what happens after a large input change.

Rank

The rank indicates how many independent output directions can be produced locally.

A rank-deficient Jacobian can indicate:

• Redundant variables
• Lost dimensions
• Local non-invertibility
• A robotic singularity
• Poorly identifiable parameters

Singular values

Singular values describe how strongly the transformation expands or contracts different directions.

Very large singular values can indicate sensitivity. Very small singular values can indicate compression or near-singularity.

Conditioning

An ill-conditioned Jacobian can make optimization unstable.

Small numerical errors may produce large changes in the calculated solution.

Sparsity

Many practical Jacobians are sparse. Each output may depend on only a small subset of inputs.

Exploiting sparsity can significantly reduce memory usage and computation time.

Differentiability

Not every software operation is differentiable.

Examples that require care include:

• Hard thresholds
• Integer conversions
• Discrete branching
• Sorting
• Index-based selection
• External service calls
• Random sampling
• Non-differentiable simulators

Some operations may need smooth approximations, surrogate gradients, or custom differentiation rules.

Best Practices

1. Define the function boundary clearly

Document:

• Which values are inputs
• Which values are outputs
• Which argument is being differentiated
• Expected input and output shapes
• Whether batching is included in the Jacobian

A Jacobian with respect to model inputs is different from a Jacobian with respect to model parameters.

2. Do not construct the full Jacobian unless necessary

For a model with one million inputs and one million outputs, the complete Jacobian would contain one trillion entries.

Frequently, the real requirement is only:

Jv

or:

vᵀJ

Use JVPs, VJPs, gradient operations, or linear operators whenever possible.

3. Choose the appropriate differentiation mode

A useful rule is:

• Use forward mode when inputs are relatively few and outputs are numerous.
• Use reverse mode when outputs are relatively few and inputs are numerous.
• Benchmark both approaches for nearly square or unusually structured problems.

JAX and PyTorch both provide forward- and reverse-mode Jacobian APIs. Their documentation recommends choosing the strategy based on the relative input and output dimensions.

4. Prefer automatic differentiation

Finite differences approximate derivatives using repeated function evaluations:

∂f/∂x ≈ [f(x + ε) - f(x)] / ε

They are useful for testing but can suffer from truncation and floating-point errors.

For production AI systems, automatic differentiation is generally more accurate and efficient.

5. Validate derivatives numerically

For small test cases, compare automatic derivatives with finite-difference approximations.

JAX provides derivative-checking utilities, and PyTorch provides gradcheck for validating analytical gradients against numerical approximations.

Derivative tests are especially important when implementing:

• Custom operators
• Custom CUDA kernels
• Custom gradient functions
• Scientific models
• Complex loss functions
• External differentiable libraries

6. Test shapes as well as values

A numerically correct result with an incorrect axis order can still cause serious bugs.

Tests should verify:

assert jacobian.shape == (number_of_outputs, number_of_inputs)

For batched data, document whether batch dimensions are included or processed independently.

7. Monitor numerical stability

Check Jacobians for:

• NaN
• Positive or negative infinity
• Extremely large norms
• Unexpected zeros
• Very poor condition numbers

Use appropriate scaling, normalization, precision, and numerically stable operations.

8. Avoid explicitly calculating determinants of large matrices

Direct determinants can overflow, underflow, or become expensive.

When a log determinant is required, prefer stable operations such as:

log |det(J)|

Use specialized matrix factorizations or APIs such as slogdet instead of calculating det(J) and then taking its logarithm.

9. Exploit structure

Look for:

• Sparse Jacobians
• Block-diagonal structure
• Triangular matrices
• Low-rank approximations
• Repeated subexpressions
• Independent batch elements

Normalizing-flow architectures often deliberately constrain the Jacobian structure so that determinant calculations remain tractable.

10. Be careful with custom gradients

A custom gradient may improve performance or numerical stability, but it can also silently introduce incorrect training behavior.

TensorFlow’s tf.custom_gradient, for example, allows developers to define specialized gradient calculations. The framework documentation recommends such control when a more efficient or numerically stable gradient is needed.

Every custom gradient should be supported by:

• Mathematical documentation
• Unit tests
• Numerical gradient checks
• Edge-case tests
• Precision tests

11. Separate analysis code from production inference

Jacobians can be expensive.

A production inference endpoint may not need to calculate them for every request. Sensitivity analysis can often run:

• During training
• In offline evaluation
• On sampled requests
• During model validation
• As part of monitoring jobs

12. Profile before optimizing

The bottleneck may be:

• Repeated forward passes
• Graph retention
• Materializing a dense Jacobian
• Excessive precision
• Poor batching
• Host-device transfers
• An inefficient forward/reverse-mode choice

Measure memory and runtime before redesigning the implementation.

Common Mistakes

Confusing a gradient with a Jacobian

A gradient normally belongs to a scalar-valued function. A Jacobian belongs to a vector-valued function.

Confusing the matrix with its determinant

The Jacobian matrix exists for rectangular mappings. Its determinant exists only for square Jacobians.

Assuming the Jacobian is constant

Only linear or affine transformations have constant Jacobians. Nonlinear models usually have input-dependent Jacobians.

Ignoring matrix orientation

Some libraries and textbooks use different row-vector and column-vector conventions.

Always verify the expected shape and multiplication order.

Building a dense Jacobian unnecessarily

Backpropagation normally needs a VJP, not the complete matrix.

Using finite differences with a poor epsilon

An epsilon that is too large creates approximation error. An epsilon that is too small creates floating-point cancellation.

Treating local sensitivity as causality

A large partial derivative shows local sensitivity under the mathematical model. It does not prove that the input causes the output in the real world.

Ignoring non-differentiable points

ReLU, maximum operations, clipping, and piecewise functions may not have a unique derivative at certain points. Frameworks typically select a defined subgradient or implementation-specific derivative.

Integrating Jacobians into a Software Development Workflow

Step 1: Identify the differentiable component

Determine which part of the application can be represented as:

output = F(input, parameters)

Step 2: Define the engineering question

Decide whether you need:

• A gradient
• A full Jacobian
• A JVP
• A VJP
• A determinant
• A log determinant
• A Jacobian norm
• Singular values
• A Hessian

Do not calculate a full Jacobian when a smaller operation answers the question.

Step 3: Select an automatic differentiation framework

Common options include:

• PyTorch
• TensorFlow
• JAX
• Automatic differentiation libraries in scientific languages
• Symbolic systems for small analytical problems

Step 4: Build a small analytical test

Create a simple function with a Jacobian that can be calculated by hand.

Use it to verify:

• Matrix orientation
• Shape conventions
• Data types
• Framework behavior

Step 5: Add derivative tests

Compare automatic differentiation with finite differences on small randomized inputs.

Step 6: Profile memory and runtime

Test realistic input and output dimensions.

A method that works for a two-dimensional example may fail when used with millions of parameters.

Step 7: Add observability

For sensitive systems, monitor statistics such as:

• Gradient norm
• Jacobian norm
• Percentage of zero derivatives
• Largest singular value
• Condition estimates
• Frequency of NaN or infinite values

Step 8: Document assumptions

Record:

• Differentiation convention
• Batch handling
• Precision
• Expected shapes
• Known non-differentiable operations
• Custom gradient behavior

Final Thoughts

The Jacobian matrix is one of the most important bridges between mathematics and modern software engineering.

It extends the derivative from a single input and output to complex systems containing many interacting variables. More importantly, it provides a local linear representation of nonlinear behavior.

In artificial intelligence, the Jacobian is behind:

• Backpropagation
• Automatic differentiation
• Input sensitivity
• Model robustness
• Neural ODEs
• Normalizing flows
• Scientific machine learning
• Gradient-based optimization

In broader software development, it supports robotics, graphics, control systems, simulations, inverse problems, and coordinate transformations.

Developers rarely need to construct an enormous Jacobian explicitly. Modern frameworks instead compute efficient Jacobian-vector and vector-Jacobian products. Understanding the matrix behind these operations, however, makes it easier to select the correct algorithm, diagnose unstable training, validate custom gradients, and design reliable differentiable systems.

The Jacobian is therefore not simply a matrix of partial derivatives. It is a practical model of how a complex system responds to change.