Quadratic Functions in Discrete-Time Control

Introduction

In discrete-time control, especially in optimal control problems like the Linear Quadratic Regulator (LQR), quadratic functions appear everywhere. Why quadratics?

In short, quadratic expressions provide a convenient "bowl-shaped" cost landscape and lead to elegant linear feedback laws. This tutorial will build up the intuition from basic calculus and linear algebra all the way to linear dynamics and optimal control.

We'll refresh key math concepts (gradients, Hessians, vectors, matrices) and then see how linear dynamics ($x_{k+1} = A\,x_k + B\,u_k$) paired with quadratic costs produce structure: value functions of the form $V(x) = x^T P x$ and optimal controls that are linear in the state.

Calculus Refresher: Gradients, Minima, and Taylor Expansion

Gradients and Local Minima

In single-variable calculus, we know a function $f(x)$ has an extremum (min or max) where $f'(x)=0$. In multiple dimensions, the gradient $\nabla f(x)$ generalizes the derivative. The gradient is a vector of all partial derivatives.

A point $x^*$ is critical (a candidate for extremum) if:

This is the multi-dimensional "flat slope" condition. However, a zero gradient alone isn't enough to ensure a minimum — it could be a maximum or a saddle point. We need to check the second derivative information next.

Hessian and Second-Order Test

The Hessian matrix $H(x)$ is the matrix of second partial derivatives (essentially the multi-dimensional second derivative). If at a critical point $x^*$ the Hessian $H(x^*)$ is positive definite, it means the function curves upward in all directions around $x^*$. This guarantees $x^*$ is a local minimum.

One way to see this is via the second-order Taylor expansion. For a function $f(x)$, around a critical point $x^*$ we have:

The linear term vanishes (gradient is zero). This quadratic approximation $\frac{1}{2}\delta^T H \delta$ tells us the local curvature. If $H$ is positive definite, then $\delta^T H \delta > 0$ for any small displacement $\delta \neq 0$, so $f(x^*+\delta) > f(x^*)$ — in other words $x^*$ is a strict local minimum. Thus positive definiteness of the Hessian is the mathematical condition for a bowl-shaped minimum.

Gradients and Steepest Descent

The gradient $\nabla f(x)$ itself points in the direction of greatest increase of $f$. For minimization, one often goes in the opposite direction (steepest descent) to go downhill on the function's surface.

We mention this because it provides intuition later: for a quadratic cost $V(x)=x^T P x$, we have $\nabla V(x) = 2Px$ which is linear in $x$, and — if $P$ defines a bowl shape — $-Px$ points toward the bowl's bottom (the origin). This "downhill direction" insight will reappear when interpreting optimal feedback controls.

Linear Algebra Foundations

Vectors and Dot Products

We assume a basic familiarity with vectors and matrices. A quick refresher: a vector $\boldsymbol{x} \in \mathbb{R}^n$ can be viewed as an $n\times 1$ column of numbers. The dot product $x^T y = \sum_{i=1}^n x_i y_i$ between two vectors yields a scalar.

Geometrically, the dot product relates to the angle between vectors and their lengths, but algebraically it's a sum-of-products. When a vector is dotted with itself, $x^T x = \sum_i x_i^2$, which is the square of its Euclidean norm (length).

Matrices and Linear Maps

A matrix $A$ is essentially a linear transformation. Multiplying $A$ by a vector $x$ (giving $Ax$) produces a new vector. If $A$ is $m\times n$, it maps $\mathbb{R}^n$ to $\mathbb{R}^m$. In control applications, matrices often describe system dynamics or quadratic cost weights.

One special thing to recall: any symmetric matrix can be diagonalized by an orthonormal basis of eigenvectors. This means if $A=A^T$, we can find an orthogonal matrix $P$ and diagonal $\Lambda$ such that $A = P^T \Lambda P$. This fact is very useful for understanding quadratic forms.

Quadratic Forms

An expression like $x^T P x$ is called a quadratic form in $x$. It outputs a scalar. If $P$ is symmetric, $x^T P x$ can be seen as a generalized "squared length" of $x$ under the metric defined by $P$.

For example, if $P = I$ (identity matrix), then $x^T P x = x^T x = |x|^2$ is just the usual square length. If $P$ is different, $x^T P x$ scales and rotates that bowl.

By using the eigen-decomposition $P = P^T \Lambda P$, one can show $x^T P x = y^T \Lambda y$ where $y = P x$ is a rotated coordinate. This means $x^T P x$ is a sum of terms $\lambda_i y_i^2$ where $\lambda_i$ are eigenvalues of $P$. Each direction is stretched by $\lambda_i$.

Positive Definite Matrices

A symmetric matrix $P$ is positive definite (PD) if $x^T P x > 0$ for all $x \neq 0$. All eigenvalues of a PD matrix are positive. PD matrices are the ones that create convex quadratic forms (bowls).

Importantly, if $P$ is PD, then $f(x)=x^T P x$ has a unique minimum at $x=0$ (since $f(x)>0$ for $x\neq 0$ and $f(0)=0$). We've essentially seen this already via Hessians: $P$ being PD is analogous to Hessian PD in an optimization context.

The surface $z = x^T P x$ is an elliptic paraboloid (a multi-dimensional bowl). Because $P$ is symmetric and PD, we can think of $x^T P x$ as a weighted sum of squares in some rotated coordinate system.

Geometric Picture — Ellipsoids

The level sets of a positive definite quadratic form are ellipsoids. For example, the set $\{x: x^T P x = 1\}$ (all points giving value 1) is an ellipsoid centered at the origin.

To visualize, consider $P$ in $\mathbb{R}^2$. If $P = I$, $\{x: x^T x = 1\}$ is the unit circle (a special case of an ellipse). If $P$ stretches space more in one direction than another, the unit-level set becomes an ellipse stretched accordingly.

In general, $x^T P x = 1$ can be rewritten via diagonalization as $\sum_i (y_i^2/c_i^2) = 1$, which is recognized as the equation of an ellipsoid. Each $c_i$ relates to an eigenvalue of $P$ (specifically $c_i = 1/\sqrt{\lambda_i}$). This means along each principal axis (eigenvector direction) of $P$, the ellipsoid extends to radius $c_i$.

Linear Dynamics in Discrete Time

Now, let's shift to dynamics. In discrete-time control, a linear system is often written as:

where $x_k \in \mathbb{R}^n$ is the state at time-step $k$, and $u_k \in \mathbb{R}^m$ is the control input (action). The matrices $A$ and $B$ determine how the state evolves: $A$ maps the current state to the next (if $u=0$), and $B$ maps the control input's influence on the next state.

This equation is a linear difference equation. If we start from some initial state $x_0$, then:

• After one step: $x_1 = A x_0 + B u_0$
• After two steps: $x_2 = A x_1 + B u_1 = A^2 x_0 + A B u_0 + B u_1$
• And so on...

Without control ($u_k \equiv 0$), the state evolves as $x_k = A^k x_0$. Depending on $A$, the state might decay to 0 (if $A$ is stable, eigenvalues less than 1 in magnitude), or might blow up or oscillate. The control inputs $u_k$ are our way to influence the trajectory — ideally to drive the state to some desired condition (often the origin in regulation problems like LQR).

Linearity is crucial because it preserves quadratic forms in a nice way. For any vectors $x, u$, the next state $Ax + Bu$ is a linear combination. If we plug a linear combination into a quadratic form, we will still get a quadratic expression (with cross terms). This will become important when we discuss how costs accumulate and how we optimize the control.

Quadratic Cost Functions in Control

In control design, we often define a cost function to measure performance. A common choice (especially in LQR problems) is a quadratic cost. For example, one might define a stage cost at time $k$ as:

where $Q$ and $R$ are symmetric weight matrices (with $Q \succeq 0$ and $R \succ 0$ so that the cost is nonnegative and convex). The scalar $x^T Q x$ penalizes state deviations (e.g., how far $x$ is from the origin or desired state) and $u^T R u$ penalizes control effort (to keep inputs reasonable). These terms are quadratic forms, hence nonnegative and bowl-shaped in their respective variables.

By summing such stage costs over time (perhaps indefinitely with a discount factor or an infinite horizon sum), we get an accumulated cost. For instance, an infinite-horizon undiscounted LQR cost from time 0 onward could be:

assuming the sum converges (which it will if the system is stabilizable under some controller). The objective is to choose a sequence of controls $\{u_0, u_1, ...\}$ that minimizes this total cost, starting from a given initial state $x_0$. This is the classic LQR problem: linear dynamics, quadratic cost.

The Value Function and Bellman's Principle

To solve an optimal control problem, it's helpful to define the value function (also called the cost-to-go function). For state $x$ at time $k$, the value function $V_k(x)$ is the minimal remaining cost from $k$ onward if we start in state $x$ and act optimally.

The famous Bellman principle of optimality says that $V_k(x)$ should satisfy a recursive relationship (the Bellman equation):

where $x_{k+1} = A x + B u$ is the next state. This equation says: the optimal cost from $x$ is the immediate cost (state and control at $k$) plus the optimal future cost from the next state, minimized over the control choice $u$ at time $k$.

If we consider the infinite-horizon time-invariant case (steady-state behavior as $k\to\infty$), the value function becomes time-independent. $V(x)$ satisfies the Bellman operator equation:

This is a functional equation: $V(x)$ must equal the minimum over $u$ of that quadratic expression. The remarkable fact for LQR is that we can guess the form of $V(x)$ and verify it.

We anticipate (or can derive) that the optimal value function will be quadratic in $x$. In particular, one can assume:

for some symmetric positive semidefinite matrix $P$ that we need to determine. This is a logical guess because if all stage costs are quadratic and dynamics are linear, it would be consistent for the accumulated cost-to-go to also form a quadratic. Indeed, it is "well known that for this problem the optimal cost-to-go function is quadratic."

Why the Value Function Becomes Quadratic (Bellman Backup)

Assume $V(x) = x^T P x$ for some matrix $P$ (we'll find $P$ by solving Bellman's equation). Now plug this form into the Bellman equation:

where we expanded $(Ax+Bu)^T P (Ax+Bu)$ into quadratic form. Notice: this is a quadratic expression in the control $u$ (for a fixed state $x$). Let's write it as:

Here $(R + B^T P B)$ is an $m \times m$ matrix and $2A^T P B\,x$ is an $n\times 1$ vector that is the linear term in $u$. Minimizing a quadratic function of $u$ is a textbook exercise: since $R$ is positive definite, and $P$ will turn out positive definite as well, $(R + B^T P B)$ is positive definite — so the quadratic in $u$ is convex.

We can find the minimizing $u$ by setting the gradient (with respect to $u$) to zero. The gradient of the cost w.r.t. $u$ is:

and setting this equal to zero yields the optimal control law:

This is a linear state-feedback law, of the form $u^* = -K x$ where $K = (R + B^T P B)^{-1} B^T P A$. We see explicitly that the minimization over $u$ produced a control that is an affine function of $x$ — in fact linear, since it goes through the origin (no constant term). This confirms the intuition that in an LQR, the optimal policy is linear in the state.

Now, having found $u^*(x)$ in terms of $P$, we can plug $u^*$ back into the cost expression to determine $P$ itself. When we substitute $u^*$, the cross-terms cancel out and we end up with a purely quadratic form in $x$:

Because this must equal $x^T P x$ for all $x$, we can identify the new $P$ inside the big parentheses. That is, $P$ should satisfy the discrete-time algebraic Riccati equation (DARE):

This Riccati equation is essentially the outcome of requiring consistency in our quadratic value function assumption. Under controllability/stabilizability conditions, there is a unique positive definite solution $P$ that yields a $K$ which stabilizes the closed-loop system.

Interpretation: Cost Bowls and Ellipsoidal Contours

Let's step back and put the pieces together with some intuition. We have a value function $V(x) = x^T P x$. You can imagine $V(x)$ as a cost landscape over the state space. It's a bowl (paraboloid) shaped by $P$.

The contours $V(x) = \text{constant}$ are ellipsoids as we discussed. If you are at state $x$, $V(x)$ tells you how much cost you'll accumulate in the future if you act optimally (it's the "optimal future cost" from $x$). Lower $V(x)$ means you're in a better region (closer to the goal, e.g. the origin), higher $V(x)$ means you're further "up the bowl" in terms of cost. The minimum is at $x=0$ (assuming we want to stabilize to zero), where $V(0)=0$ — no cost from the origin if you're already there and stay there.

Now, the optimal control law we found, $u^* = -K x$, can be viewed in this landscape perspective: it is the controller that tries to drive the state downhill on the cost bowl as fast as possible.

Since $\nabla V(x) = 2 P x$, a steepest descent direction in state space would be $-\nabla V(x) = -2Px$ (pointing toward lower $V$). The optimal controller's action $-Kx$ is essentially designed to push the state in that general direction.

In fact, in the special case where $B$ can command all directions equally and $R$ is identity, the formula simplifies and $K$ aligns with $P$ such that $-Kx = -R^{-1}B^T P x = -B^T P x$ (when $R=I$ and if $B$ is square invertible, $K = B^T P$). Then $u^* = -B^T P x$ literally uses $B^T P x$, which is proportional to $Px$, as the direction to push — effectively the projection of the steepest descent direction $-Px$ onto the control inputs.

This geometric viewpoint is powerful. The matrix $P$ in $V(x)=x^T P x$ can be interpreted as defining "cost contours" — an ellipsoid $x^T P x = c$ is like a confidence ellipsoid of equivalent cost. The optimal control law $u=-Kx$ will always try to move the state to lower ellipsoids (closer to the center).

In LQR, these ellipsoids also often represent something like Lyapunov level sets for the closed-loop system: $V(x)$ can serve as a Lyapunov function proving stability of $x_{k+1}=(A-BK)x_k$. Indeed, solving the Riccati equation for $P$ is fundamentally related to finding a Lyapunov function $V(x)=x^T P x$ that satisfies the optimal cost decrease along trajectories.

Closing Thoughts

We started from basic calculus and linear algebra, and we've arrived at the core structure of the discrete-time LQR: quadratic costs produce quadratic value functions and linear optimal feedback laws. Let's recap the conceptual insights: