Image reconstruction by domain-transform manifold learning

1 Letter Image reconstruction by domain-transform manifold learning Bo Zhu1,2,3, Jeremiah Z. Liu4, Stephen F. Cauley1,2, Bruce R. Rosen1,2 & Matthew S. Rosen1,2,3. Image reconstruction is essential for imaging applications across Inspired by the perceptual learning archetype, we describe here the physical and life sciences, including optical and radar systems, a data-driven unified Image reconstruction approach, which we magnetic resonance imaging, X-ray computed tomography, call AUTOMAP, that learns a reconstruction mapping between the positron emission tomography, ultrasound imaging and radio sensor-domain data and Image -domain output (Fig.)

2 1a). As this map- astronomy1 3. During Image acquisition, the sensor encodes an ping is trained, a low-dimensional joint manifold of the data in both intermediate representation of an object in the sensor domain, domains is implicitly learned (Fig. 1b), capturing a highly expressive which is subsequently reconstructed into an Image by an inversion representation that is robust to noise and other input perturbations. of the encoding function. Image reconstruction is challenging We implemented the AUTOMAP unified reconstruction framework because analytic knowledge of the exact inverse transform may not with a deep neural network feed-forward architecture composed of exist a priori, especially in the presence of sensor non-idealities fully connected layers followed by a sparse convolutional autoencoder and noise.

3 Thus, the standard reconstruction approach involves (Fig. 1c). The fully connected layers approximate the between- manifold approximating the inverse function with multiple ad hoc stages in projection from the sensor domain to the Image domain. The convo- a signal processing chain4,5, the composition of which depends on lutional layers extract high-level features from the data and force the the details of each acquisition strategy, and often requires expert Image to be represented sparsely in the convolutional-feature space. parameter tuning to optimize reconstruction performance.

4 Here Our network operates similarly to the denoising autoencoder described we present a unified framework for Image reconstruction previously10, but rather than finding an efficient representation of the automated transform by manifold approximation (AUTOMAP) identity to map f (x ) = x x 1(x ) = x over the manifold of inputs X. which recasts Image reconstruction as a data-driven supervised (where x maps the intrinsic coordinate system of X to Euclidean space learning task that allows a mapping between the sensor and the near x), AUTOMAP determines both a between- manifold projection Image domain to emerge from an appropriate corpus of training g from X (the manifold of sensor inputs) to Y (the manifold of output data.)

5 We implement AUTOMAP with a deep neural network and images), and a manifold mapping y to project the Image from manifold exhibit its flexibility in learning reconstruction transforms for Y back to the representation in Euclidean space. A composite inverse various magnetic resonance imaging acquisition strategies, using transformation f (x ) = y g x 1(x ) over the joint manifold MX,Y =. the same network architecture and hyperparameters. We further X Y (Fig. 1b) is achieved. A full mathematical description of this demonstrate that manifold learning during training results in manifold learning process is detailed in Methods.

6 Sparse representations of domain transforms along low-dimensional In contrast to previous efforts that use neural networks to solve data manifolds, and observe superior immunity to noise and a inverse functions11 13, our approach searches for an inverse that best reduction in reconstruction artefacts compared with conventional represents the data in a low-dimensional feature space determined by handcrafted reconstruction methods. In addition to improving the manifold learning as well as the trained sparse convolutional filters. reconstruction performance of existing acquisition methodologies, Furthermore, AUTOMAP solves a generalized reconstruction problem we anticipate that AUTOMAP and other learned reconstruction and thus differs from work using neural networks to implement a approaches will accelerate the development of new acquisition specific Image reconstruction task14 17.

7 These previous approaches strategies across imaging modalities. use known properties of the canonical domain transform to formulate The paradigm shift from manual to automatic feature extraction in a the neural network model, or perform the explicit transform before host of machine learning tasks including speech recognition6 and Image processing by a neural network used for Image -space artefact reduction. classification7 has demonstrated the advantage of allowing real-world We demonstrate AUTOMAP Image reconstruction using MRI as data to guide efficient representation through a structured training a model system, but we emphasize that our approach is applicable to process.

8 This strategy is mirrored in biological organisms for refining Image reconstruction problems across a broad range of modalities visual perception in a process known as perceptual learning8. Human given the mathematical similarities of tomographic spatial encoding visual reconstruction of time-domain neural codes into the percept functions typically governed by Fredholm integral equations1. The Image is trained through experience during cognitive development plethora of MRI acquisition strategies makes it a particularly appro- into adulthood. This conditioning on prior data has been shown to be priate platform to exhibit the flexibility of AUTOMAP reconstruction critical to robust performance in low signal-to-noise settings9, which over a variety of encoding schemes.

9 We first evaluated the performance are fundamentally challenging for artificial imaging systems across dis- of AUTOMAP alongside conventional methods in four nontrivial ciplines and applications. In contemporary medical imaging, faithful reconstruction tasks: (1) Radon projection imaging and model-based reconstruction of noisy Image acquisitions is of particular importance iterative reconstruction4; (2) spiral-trajectory k-space (rapid acquisi- as the clinical push for faster scanning increasingly relies on acquisition tion with non-Cartesian sampling) and conjugate- gradient sensitivity strategies that result in a reduction of the signal-to-noise ratio, be they encoding (SENSE) reconstruction employing non-uniform fast undersampled magnetic resonance imaging (MRI), or low-dose X-ray Fourier transform (NUFFT) regridding5; (3) Poisson-disk undersam- computed tomography imaging.

10 Pled k-space (incoherent sparse acquisition) and compressed sensing 1. A. A. Martinos Center for Biomedical Imaging, Massachusetts General Hospital, Boston, Massachusetts, USA. 2 Harvard Medical School, Boston, Massachusetts, USA. 3 Department of Physics, Harvard University, Cambridge, Massachusetts, 4 Department of Biostatistics, Harvard University, Cambridge, Massachusetts, USA. 2 2 m a rc h 2 0 1 8 | VO L 5 5 5 | NAT U R E | 4 8 7. 2018 Macmillan Publishers Limited, part of Springer Nature. All rights reserved. RESEARCH Letter Ix 1. b in ma a do Conventional Sensor reconstruction chain Image or x ns g Se Cartesian MRI.