Derive the Jacobian and the Hessian of $ \phi(\boldsymbol X)=\left \lVert{\boldsymbol X \boldsymbol v}\right \rVert $

$\def\half{\frac{1}{2}} \def\bm#1{\boldsymbol{#1}} \def\R{\mathbb{R}} \def\norm#1{\left\lVert #1 \right\rVert} \def\X{\bm{X}} \def\v{\bm{v}} \def\vec{\operatorname{vec}} \def\d{{\sf d}} \def\Xv{\widehat{\X\v}} \def\D{\operatorname{D}} \def\H{\operatorname{H}} \def\It{{\bm I}_2}$$\textbf{Exercise}$: Derive the Jacobian and the Hessian of $$ \phi(\boldsymbol X)=\norm{\boldsymbol X \boldsymbol v} $$ w.r.t. $\bm{X}$ where $\bm{X} \in \R^{2 \times N}$ and $\bm{v} \in \R^{N \times 1}$.

$\textit{Solution}$. Since $$\norm{\X\v}=\qty(\v^\top\X^\top\X\v)^{\half}$$ we have \begin{align*} \d\phi & = \half \qty(\v^\top\X^\top\X\v)^{-\half} \d\qty(\v^\top\X^\top\X\v) \\ & = \half \norm{\X\v}^{-1} \qty(\v^\top\qty(\d\X)^\top\X\v+\v^\top\X^\top\qty(\d\X)\v) \\ & = \half \norm{\X\v}^{-1} 2 \v^\top\X^\top\qty(\d\X)\v & {}^* \tag{I} \end{align*} ${}^*$ $\bm a^\top \bm b = \bm b^\top \bm a$
If we apply the $\vec$ operator to both sides of the equation and use the property $$ \vec\bm a^\top \X \bm b = \qty(\bm b^\top \otimes \bm a^\top) \vec\X $$ we obtain \begin{equation*} \d\phi=\norm{\X\v}^{-1} \qty(\v^\top \otimes \qty(\X\v)^\top) \d\vec\X. \end{equation*} Therefore the Jacobian is $$ \D\phi(\X)=\pdv{\phi(\X)}{\qty(\vec\X)^\top}=\norm{\X\v}^{-1} \qty(\v^\top \otimes \qty(\X\v)^\top). $$ To derive the Hessian start from ($\text{I}$) and apply the differential operator: \begin{align*} \d^2\phi & = \norm{\X\v}^{-1} \qty(\v^\top\d\X^\top\qty(\d\X)\v) -\half \norm{\X\v}^{-3} \d\qty(\v^\top\X^\top\X\v) \qty(\v^\top\X^\top\qty(\d\X)\v) & {}^{1}\\ & = \norm{\X\v}^{-1} \qty(\v^\top\d\X^\top\qty(\d\X)\v) - \norm{\X\v}^{-3} \qty(\v^\top\qty(\d\X)^\top\X\v)\qty(\v^\top\X^\top\qty(\d\X)\v) &{}^{2} \\ & = \norm{\X\v}^{-1} \tr\qty(\v\v^\top\d\X^\top\d\X) - \norm{\X\v}^{-3} \tr\qty(\v\v^\top\qty(\d\X)^\top\X\v\qty(\X\v)^\top\d\X) & {}^{3} \\ & = \norm{\X\v}^{-1} \tr\qty(\v\v^\top\d\X^\top\It\d\X) - \norm{\X\v}^{-3} \tr\qty(\v\v^\top\qty(\d\X)^\top\X\v\qty(\X\v)^\top\d\X) \\ & = \norm{\X\v}^{-1} \qty(\vec\d\X)^\top \qty(\qty(\v\v^\top)\otimes\It)(\vec\d\X) \\ &\phantom{xxxxxx}- \norm{\X\v}^{-3} \qty(\vec\d\X)^\top \qty(\qty(\v\v^\top)\otimes\qty(\X\v\qty(\X\v)^\top)^\top)(\vec\d\X) & {}^{4} \\ & = \norm{\X\v}^{-1} \qty(\d\vec\X)^\top \qty(\qty(\v\v^\top)\otimes\It)(\d\vec\X) \\ &\phantom{xxxxxx}- \norm{\X\v}^{-1} \qty(\d\vec\X)^\top \qty(\qty(\v\v^\top)\otimes\qty(\Xv\qty(\Xv)^\top))(\d\vec\X) & {}^{5} \\ & = \norm{\X\v}^{-1} \qty(\d\vec\X)^\top \qty(\qty(\v\v^\top)\otimes\qty(\It - \qty(\Xv\qty(\Xv)^\top))) \d\vec\X & {}^{6} \end{align*} ${}^1$ $\d\norm{\X\v}^{-1}=\d\qty(\v^\top\X^\top\X\v)^{-\half} =-\half\qty(\v^\top\X^\top\X\v)^{-\frac{3}{2}} \d\qty(\v^\top\X^\top\X\v)$
${}^2$ $\d\qty(\v^\top\X^\top\X\v)=\v^\top\qty(\d\X)^\top\X\v+\v^\top\X^\top\qty(\d\X)\v =\v^\top\qty(\d\X)^\top\X\v+\v^\top\qty(\d\X)^\top\X\v$
${}^3$ $\bm a^\top \bm b = \tr \bm b \bm a^\top$
${}^4$ $\tr\bm A \bm B \bm C \bm D = \qty(\vec \bm D^\top)^\top\qty(\bm C^\top \otimes \bm A)\vec \bm B = \qty(\vec \bm D)^\top \qty(\bm A \otimes \bm C^\top) \vec\bm B^\top$
${}^{5}$ $\widehat{\X\v}=\frac{\X\v}{\norm{\X\v}}$

The Hessian must be symmetric; since $\qty(\bm A \otimes \bm B)^\top=\bm A^\top \otimes \bm B^\top$ and $\bm I_2$, $\Xv\qty(\Xv)^\top$, $\v\v^\top$ are symmetric matrices $$ \qty(\qty(\v\v^\top)\otimes\qty(\It - \qty(\Xv\qty(\Xv)^\top)))^\top =\qty(\qty(\v\v^\top)\otimes\qty(\It - \qty(\Xv\qty(\Xv)^\top))) $$ and so $$ \H\phi(\X)= \frac{\partial^2 \phi}{\partial\qty(\vec\X)\partial\qty(\vec\X)^\top} =\norm{\X\v}^{-1} \qty(\qty(\v\v^\top)\otimes\qty(\It - \qty(\Xv\qty(\Xv)^\top))). $$

Comments

Post a Comment

Popular posts from this blog

CSAT 1

Image
Test Section A 1. You have a card of 10cm by 10cm. What is the largest volume in cm 3 of a box (without a lid) that can be obtained by cutting out a square of side x from each corner and then folding the flaps up? We can obtain an idea of the solution by looking at the following diagram: The box will have a base (marked in red on the diagram) with area $(l-2x)^2$ and height $x$. Therefore the volume of this box will be \begin{equation*} V=(l-2x)^x. \end{equation*} To maximize the volume first calculate \[ \dv{V}{x}=-4(l-2x)x+(l-2x)^2 \] using the product rule $d(u\cdot v)=du \cdot v + u \cdot dv$. Solve for $\dv*{V}{x}=0$ to find a (potential) maximum. If we assume that $x\neq l$ (a reasonable assumption since $x=l/2$ creates a nothing box) then \begin{align*} -4(l-2x)x+(l-2x)^2& =0 \\ l-2x-4x & = 0 \\ x & = \frac{l}{6}. \end{align*} To check that this is i...
Read more

Poor man's Daletskii-Krein

Image
Consider a second-order, symmetric tensor $A$ whose components $\def\R{\mathbb{R}}{A^j}_i \in \R$. We know that such a tensor has an eigen decomposition \begin{equation} \label{eigendecomp} A = V \Lambda V^\top \end{equation} where $V$ is an orthonormal, second-order tensor and ${\Lambda^j}_i=\lambda^j \delta^j_i$ ($\lambda^i$ is the $i$th eigenvalue of $A$). One corollary of (\ref{eigendecomp}) is \begin{equation} \label{An} A^n = V \Lambda^n V^\top . \end{equation} A function $f:\R \to \R$ that is analytic at $x=0$ is equal to its Maclaurin series \begin{equation} \label{maclaurin} f(x) = \sum_{n=0}^\infty \frac{f^{(n)}(0)}{n!} x^n \end{equation} where $f^{(0)}(0)=f(0)$ and $f^{(n)}(0)=\eval{\dv[n]{f}{x}}_{x=0}$. If we define \begin{equation} \label{def-fA} f(A):= \sum_{n=0}^\infty \frac{f^{(n)}(0)}{n!} A^n \end{equation} then $f(A)$ is also a second-order, sy...
Read more

PAT 2022

Image
1. What is the total resistance of the circuit ? 2. For which values of $x$ is $\qty(24-14x -3x^2)^{-1}$ positive? The $-1$ exponent does not change the sign. We can factorise $24-14x -3x^2$ as $(-3x+4)(x+6)$ which is positive either if $x < \frac{4}{3}, x > -6$ or if $x > \frac{4}{3}$, $x < -6$. The second condition cannot hold so we have to go with (D). 3. Molecules of oxygen in the atmosphere absorb solar radiation in bands centred at about 80 nm, 650 nm and 1000 nm. In which parts of the electromagnetic spectrum are these absorption bands? Picture from tikz.net The visible spectrum is in the very narrow band from 380 nm (violet) to 750 nm (red) so 650 nm is in the visible spectrum in the red zone. $1000 \ {\rm nm}=10^{-6} \ {\rm m}$ which is right at the edge of infrared. Finally $80 \ {\rm nm}= 0...
Read more