\( \newcommand{\A}{\mathcal{A}} \newcommand{\E}{\mathcal{E}} \newcommand{\N}{\mathcal{N}} \newcommand{\X}{\mathcal{X}} \newcommand{\ip}[2]{\langle #1, #2\rangle} \)
The theory of unitary invariants for nests was developed by Erdos. See also Davidson Chapter 7.
Proposition 1. Let \(\A\) be an abelian von Neumann algebra on a separable Hilbert space \(\H\). Then \(\A'\) has a cyclic vector, which, as a consequence, is also a separating vetor for \(\A\).
Definition 2. A nest is a linearly ordered set of projections in \(B(\H)\) which is order-complete (closed under arbitrary meets and joins) and contains \(0\) and \(I\).
Properties of nests:
Erdos developed the theory for arbitrary Hilbert spaces, but the treatment becomes much simpler if we restrict to separable Hilberts spaces, which we shall do throughout this page.
Let \(x_0\in\H\) be a fixed unit separating vector for \(\N''\). The map \[ \alpha: N\in\N \longmapsto \ip{Nx_0}{x_0}\in[0,1] \] is continuous, increasing, one-to-one, and maps \(0\) to \(0\) and \(I\) to \(1\). Thus \(\X := \alpha(\N)\) is a compact subset of \([0, 1]\) and \(\alpha^{-1}\) is continuous. Parameterize \(\N\) by \(N_t := \alpha^{-1}(t)\) for \(t\in\X\). We can easily extend the parameterization to the whole interval \([0,1]\) by defining \[ N_t := \bigvee\{N_s : s \le t, s\in\X \} \] This parameterization is monotonic and right-continuous.
Let \(x\in\H\). The function \(g_x(t) = \ip{N_t x}{x}\) is monotonic and right-continuous on \(\X\). Let \(\mu_x\) be the Lebesgue-Stieltjes measure on \([0,1]\) obtained from \(g_x\). Next, for \(x, y\in\H\) define a complex measure using the polarization identity \[ \mu_{x,y}(S) = \sum_{k=0}^3 i^k \mu_{x + i^ky}(S) \] It follows that for \(t\in\X\), \[ \begin{align} \mu_{x,y}((0,t]) &= \sum_{k=0}^3 i^k \mu_{x + i^ky}([0,t]) \\ &= \sum_{k=0}^3 i^k g_{x + i^ky}(t) - g_{x + i^ky}(0) \\ &= \sum_{k=0}^3 i^k \ip{N_t(x + i^ky)}{x + i^ky} \\ &= \ip{N_t x}{y} \end{align} \] Next, one verifies that for any fixed Borel set \(S\), \((x,y)\mapsto\mu_{x,y}(S)\) is a bounded bilinear form. This follows because the set of \(S\) for which this is true is a \(\sigma\)-algebra and contains all the half-open intervals \((a, b]\). Thus for each Borel set \(S\) there is a bounded operator \(\E(S)\) such that \[ \mu_{x,y}(S) = \ip{\E(S)x}{y} \]
Proposition 3. The map \(S\mapsto \E(S)\) is a projection-valued measure.
Proof. Or, at least the idea of a sketch of it...
Lemma 4. If \(\N\) is a continuous nest then \(\E\) is equivalent to Lebesgue measure on the Borel sets of \([0,1]\).
Proof. Since \(\N\) is continuous, \(\X\) (the range of \(\alpha\), defined above), is equal to \([0,1]\) and for any \(t\in[0,1]\) \[ \begin{align} \ip{\E((0, t])x_0}{x_0} &= \ip{N_t x_0}{x_0} \\ &= \ip{\alpha^{-1}(t) x_0}{x_0} \\ &= \alpha(\alpha^{-1}(t)) \\ &= t \\ &= m((0, t]) \end{align} \] It follows that \(\ip{\E(S)x_0}{x_0} = m(S)\) for any Borel set \(S\) and so, since \(x_0\) is a separating vector for \(\N''\) (which contains all \(\E(S)\)), \(\E(S)=0\) if and only if \(m(S)=0\).
Definition 5. A nest \(\N\) is said to be multiplicity-free if \(\N''\) has a cyclic vector.
Theorem 6. All multiplicity-free continuous nests are unitarily equivalent.
Proof. It suffices to show that an arbitrary multiplicity-free continuous nest \(\N\) is unitarily equivalent to the Volterra nest on \(L^2([0,1])\). Since \(\N\) is multiplicity-free, \(\A := \N''\) has a cyclic vector \(x_0\), But then \(x_0\) is separating for \(\A'\), which contains \(\N''\), and so \(x_0\) is cyclic and separating for \(\N''\). Let \(\E\) be the spectral measure on \([0,1]\) described above, using \(x_0\) and the parameterization of \(\N\) derived from it.
Now for any \(f\in L^\infty([0,1])\), \[ \begin{align} \left\| \left(\int f(t) d\E(t)\right) x_0 \right\|^2 &= \left\langle \left(\int |f(t)|^2 d\E(t)\right) x_0, x_0\right\rangle \\ &= \int |f(t)|^2 d \langle \E(t) x_0, x_0\rangle \\ &= \int |f(t)|^2 dm(t) \\ &= \|f\|^2_2 \end{align} \] Thus the map \[ f \longmapsto \int f(t) dE(t) \] is well-defined and isometric from a dense subset of \(L^2([0,1])\) (i.e. \(L^\infty([0,1])\)) into \(\H\) and its range contains \(\A x_0\), which is dense in \(\H\). Thus it extends by continuity to a unitary \(U: L^2([0,1]) \rightarrow \H\). Next, for \(f\in L^2([0,1])\), any fixed \(N_t\) is equal to \(\E((0, t])) = \int \chi_{(0,t])} d\E\) and so \[ \begin{align} N_t Uf &= \int \chi_{(0,t])}d\E \; \int f d\E \\ &= \int \chi_{(0,t])}f d\E \\ &= U M_{\chi_{(0,t])}} f \end{align} \] Thus \(U^* N_t U\) is the multiplication operator \(M_{\chi_{(0,t])}}\) in \(L^2([0, 1])\).